2019-10-17T20:28:03.008 INFO:root:teuthology version: 1.0.0-d8332ea 2019-10-17T20:28:03.009 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.ruOolU.tmp -- --verbose --lock --description rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/readwrite.yaml} --name sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi --block --owner scheduled_sjust@teuthology --archive /home/teuthworker/archive/sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi/4418197 2019-10-17T20:28:03.041 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-17T20:28:03.097 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi/4418197 branch: wip-sjust-testing2 description: rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/readwrite.yaml} email: null first_in_suite: false job_id: '4418197' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-sjust-testing2 ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 5000 ms type: async mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd initial require min compat client: luminous mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd debug verify cached snaps: true osd debug verify missing on start: true osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true osd sloppy crc: true osd_discard_disconnected_ops: false crush_tunables: optimal fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 workunit: branch: wip-sjust-testing2 sha1: f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 owner: scheduled_sjust@teuthology priority: 90 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 suite: rados suite_branch: wip-sjust-testing2 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 tasks: - install: null - ceph: null - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.4309 2019-10-17T20:28:03.097 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa; will attempt to use it 2019-10-17T20:28:03.098 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa/tasks 2019-10-17T20:28:03.098 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-17T20:28:03.099 INFO:teuthology.task.internal:Checking packages... 2019-10-17T20:28:03.112 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5' 2019-10-17T20:28:03.112 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-17T20:28:03.112 INFO:teuthology.packaging:ref: None 2019-10-17T20:28:03.113 INFO:teuthology.packaging:tag: None 2019-10-17T20:28:03.113 INFO:teuthology.packaging:branch: wip-sjust-testing2 2019-10-17T20:28:03.113 INFO:teuthology.packaging:sha1: f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 2019-10-17T20:28:03.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-sjust-testing2 2019-10-17T20:28:03.264 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:28:03.265 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-17T20:28:03.279 INFO:teuthology.task.internal:no buildpackages task found 2019-10-17T20:28:03.279 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-17T20:28:03.306 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-17T20:28:03.306 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-17T20:28:03.349 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-10-17T20:28:13.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:28:23.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:28:33.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:28:43.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:28:53.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:29:03.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:29:13.449 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:29:23.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:29:33.486 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:29:43.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:29:53.513 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:30:03.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:30:13.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:30:23.563 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:30:33.580 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-10-17T20:30:43.593 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_sjust@teuthology', 'description': '/home/teuthworker/archive/sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi/4418197', 'machine_type': 'smithi'} 2019-10-17T20:30:43.608 DEBUG:teuthology.lock.ops:locked ubuntu@smithi092.front.sepia.ceph.com, ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:30:43.736 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-17T20:30:43.926 INFO:teuthology.provision.fog.smithi092:Scheduling deploy of ubuntu 18.04 2019-10-17T20:30:43.929 INFO:teuthology.provision.fog.smithi204:Scheduling deploy of ubuntu 18.04 2019-10-17T20:30:44.211 INFO:teuthology.orchestra.console:Power off smithi092 2019-10-17T20:30:44.211 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-17T20:30:44.249 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:44.303 INFO:teuthology.orchestra.console:Power off smithi204 2019-10-17T20:30:44.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-17T20:30:44.348 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:44.443 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2019-10-17T20:30:44.492 INFO:teuthology.orchestra.console:Power off for smithi204 completed 2019-10-17T20:30:44.545 INFO:teuthology.orchestra.console:Power on smithi092 2019-10-17T20:30:44.545 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-17T20:30:44.589 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:44.618 INFO:teuthology.orchestra.console:Power on smithi204 2019-10-17T20:30:44.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-17T20:30:44.658 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:48.648 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:48.712 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:48.943 INFO:teuthology.orchestra.console:Power on for smithi204 completed 2019-10-17T20:30:49.043 INFO:teuthology.provision.fog.smithi204:Waiting for deploy to finish 2019-10-17T20:30:52.780 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-17T20:30:53.107 INFO:teuthology.orchestra.console:Power on for smithi092 completed 2019-10-17T20:30:53.208 INFO:teuthology.provision.fog.smithi092:Waiting for deploy to finish 2019-10-17T20:33:49.955 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:33:59.021 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:08.078 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:09.134 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:17.133 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:18.191 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:26.197 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:27.275 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:35.281 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:36.330 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:44.365 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:45.386 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:53.420 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:34:54.442 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:02.477 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:03.498 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:12.561 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:21.646 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:33.949 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:35:39.955 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:39.965 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:35:45.975 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:45.986 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:35:51.995 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:52.005 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:35:53.153 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:35:58.024 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:58.036 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:35:59.159 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:35:59.167 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:36:04.041 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:04.128 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.128 INFO:teuthology.orchestra.run.smithi204:> hostname 2019-10-17T20:36:04.502 INFO:teuthology.orchestra.run.smithi204.stdout:smithi204.front.sepia.ceph.com 2019-10-17T20:36:04.503 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.503 INFO:teuthology.orchestra.run.smithi204:> grep smithi204.front.sepia.ceph.com /etc/hosts 2019-10-17T20:36:04.574 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.15.204 smithi204.front.sepia.ceph.com smithi204 2019-10-17T20:36:04.575 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.575 INFO:teuthology.orchestra.run.smithi204:> sudo hostname smithi204 2019-10-17T20:36:04.773 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.773 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/smithi204.front.sepia.ceph.com/smithi204/g' /etc/hosts 2019-10-17T20:36:04.785 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.785 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/172.21.15.204/172.21.15.204/g' /etc/hosts 2019-10-17T20:36:04.868 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.868 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/smithi204.front.sepia.ceph.com/smithi204/g' /etc/hostname 2019-10-17T20:36:04.945 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:04.946 INFO:teuthology.orchestra.run.smithi204:> sudo hostname smithi204 2019-10-17T20:36:05.023 INFO:teuthology.provision.fog.smithi204:Deploy complete! 2019-10-17T20:36:05.026 DEBUG:teuthology.parallel:result is None 2019-10-17T20:36:05.169 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:05.179 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:36:11.189 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:11.198 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:36:17.208 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:17.218 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-10-17T20:36:23.228 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:23.312 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:23.313 INFO:teuthology.orchestra.run.smithi092:> hostname 2019-10-17T20:36:23.685 INFO:teuthology.orchestra.run.smithi092.stdout:smithi092.front.sepia.ceph.com 2019-10-17T20:36:23.685 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:23.685 INFO:teuthology.orchestra.run.smithi092:> grep smithi092.front.sepia.ceph.com /etc/hosts 2019-10-17T20:36:23.754 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.92 smithi092.front.sepia.ceph.com smithi092 2019-10-17T20:36:23.754 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:23.754 INFO:teuthology.orchestra.run.smithi092:> sudo hostname smithi092 2019-10-17T20:36:23.898 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:23.898 INFO:teuthology.orchestra.run.smithi092:> sudo sed -i -e 's/smithi092.front.sepia.ceph.com/smithi092/g' /etc/hosts 2019-10-17T20:36:23.913 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:23.913 INFO:teuthology.orchestra.run.smithi092:> sudo sed -i -e 's/172.21.15.92/172.21.15.92/g' /etc/hosts 2019-10-17T20:36:24.008 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:24.008 INFO:teuthology.orchestra.run.smithi092:> sudo sed -i -e 's/smithi092.front.sepia.ceph.com/smithi092/g' /etc/hostname 2019-10-17T20:36:24.088 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:24.088 INFO:teuthology.orchestra.run.smithi092:> sudo hostname smithi092 2019-10-17T20:36:24.165 INFO:teuthology.provision.fog.smithi092:Deploy complete! 2019-10-17T20:36:24.168 DEBUG:teuthology.parallel:result is None 2019-10-17T20:36:24.330 INFO:teuthology.lock.keys:Checking smithi092.front.sepia.ceph.com 2019-10-17T20:36:24.331 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-17T20:36:24.345 INFO:teuthology.lock.keys:Checking smithi204.front.sepia.ceph.com 2019-10-17T20:36:24.346 INFO:teuthology.lock.keys:New key found. Updating... 2019-10-17T20:36:24.360 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: set os type and version on server 2019-10-17T20:36:24.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:36:24.361 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:24.444 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:24.444 INFO:teuthology.orchestra.run.smithi092:> true 2019-10-17T20:36:24.615 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:24.616 INFO:teuthology.orchestra.run.smithi092:> uname -m 2019-10-17T20:36:24.686 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2019-10-17T20:36:24.686 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:24.686 INFO:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2019-10-17T20:36:24.738 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2019-10-17T20:36:24.738 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-17T20:36:24.738 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2019-10-17T20:36:24.738 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2019-10-17T20:36:24.738 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="18.04" 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=bionic 2019-10-17T20:36:24.739 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=bionic 2019-10-17T20:36:24.740 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2019-10-17T20:36:24.752 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: set os type and version on server 2019-10-17T20:36:24.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:36:24.752 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:24.828 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:24.828 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-17T20:36:25.042 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:25.042 INFO:teuthology.orchestra.run.smithi204:> uname -m 2019-10-17T20:36:25.111 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2019-10-17T20:36:25.112 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:25.112 INFO:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2019-10-17T20:36:25.181 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2019-10-17T20:36:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-17T20:36:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2019-10-17T20:36:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2019-10-17T20:36:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-17T20:36:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="18.04" 2019-10-17T20:36:25.182 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-17T20:36:25.183 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-17T20:36:25.183 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-17T20:36:25.183 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-17T20:36:25.183 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=bionic 2019-10-17T20:36:25.183 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=bionic 2019-10-17T20:36:25.184 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2019-10-17T20:36:25.196 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-10-17T20:36:25.222 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvZvqdwAiqPH7Tv1SjnaVZbTB7C3O9xmcp8DXENfY2xFXll6gLqfr1/nmtdlJqRuSM68CE+wiFlpv/97UuOtY5a99TYfOzNqZL8qJxpfNs9wH12XDF2Jjyd9rLVwrH9pcARm4icefXOTSlntJt0YdOn/IJbjeeUulmAxLtMYzM/t4BD0TPrAi061MNPZamx99gWEZwZkN3FFTWHGNR+hKaZUqwSoFKGVxuSncL/Y/FJ27itleZRQTHTdPDbjeLe6e6yvgf/pLyAu/AvnIqrSyHBY+khT/2mYNqwfXY5mwZIBQRRGOI1RgRDxWuX7mp3oI5aCSE6BMFjw/NA5Dbv1RN smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxcgkWcgpyXJpHPdcMl8MDrjKlUgM3Us9wefFAGIALtC4e2S4bWyY/L9DXlacl7CJCIIhy6oB7yzN/KZ35HJskpcwp6GAWk0PMRHnhOTC5xioN5fskgms3DKh5UXWqb9Ulyn4UWNRkKv20J5Lr43Jfo0qZ9QlSLRVf+IrVLturRG49z2bZYvi2yapEuI28/HezE9qpx5Hd1L5qWQ4Fmjpz3XVjrjMgDduhjCEbyuQj5xGp8xFREZB1fYTfJ6JocCgqCfxH58cWTEB+c2ZFxV5HFrN9/VWLTtWlVaS6N4qNNmy1uQh/qUA94Jqb02P8Svi2/7YwIsYBhixu6JCBvGu9 2019-10-17T20:36:25.222 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-17T20:36:25.265 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-10-17T20:36:25.273 INFO:teuthology.task.internal:Saving configuration 2019-10-17T20:36:25.293 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-10-17T20:36:25.308 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-10-17T20:36:25.321 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-17 20:30:43.599047', u'locked_by': u'scheduled_sjust@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi092.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJABoIeQJwUd7aDRWaC+IV2a6JyxfGl/fN40EIrgxTGT', u'description': u'/home/teuthworker/archive/sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi/4418197'} 2019-10-17T20:36:25.334 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-10-17 20:30:43.599027', u'locked_by': u'scheduled_sjust@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi204.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF8H48K0MbVIg4Oyt7fLXDjrNI4nR94y//Z98RLeF6Om', u'description': u'/home/teuthworker/archive/sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi/4418197'} 2019-10-17T20:36:25.334 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-10-17T20:36:25.341 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-10-17T20:36:25.341 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2019-10-17T20:36:25.341 INFO:teuthology.run_tasks:Running task console_log... 2019-10-17T20:36:25.471 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-17T20:36:25.472 INFO:teuthology.run_tasks:Running task internal.connect... 2019-10-17T20:36:25.496 INFO:teuthology.task.internal:Opening connections... 2019-10-17T20:36:25.496 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:36:25.497 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:25.581 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:36:25.581 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:36:25.664 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-10-17T20:36:25.674 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:25.674 INFO:teuthology.orchestra.run.smithi092:> uname -m 2019-10-17T20:36:25.744 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2019-10-17T20:36:25.744 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:25.744 INFO:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2019-10-17T20:36:25.814 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Ubuntu" 2019-10-17T20:36:25.814 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-17T20:36:25.815 INFO:teuthology.orchestra.run.smithi092.stdout:ID=ubuntu 2019-10-17T20:36:25.815 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE=debian 2019-10-17T20:36:25.815 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="18.04" 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_CODENAME=bionic 2019-10-17T20:36:25.816 INFO:teuthology.orchestra.run.smithi092.stdout:UBUNTU_CODENAME=bionic 2019-10-17T20:36:25.817 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2019-10-17T20:36:25.830 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:25.830 INFO:teuthology.orchestra.run.smithi204:> uname -m 2019-10-17T20:36:25.851 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2019-10-17T20:36:25.851 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:25.851 INFO:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2019-10-17T20:36:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu" 2019-10-17T20:36:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2019-10-17T20:36:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu 2019-10-17T20:36:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian 2019-10-17T20:36:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2019-10-17T20:36:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="18.04" 2019-10-17T20:36:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/" 2019-10-17T20:36:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2019-10-17T20:36:25.922 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2019-10-17T20:36:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2019-10-17T20:36:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=bionic 2019-10-17T20:36:25.923 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=bionic 2019-10-17T20:36:25.923 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2019-10-17T20:36:25.934 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-10-17T20:36:25.983 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-10-17T20:36:25.996 INFO:teuthology.task.internal:Checking for old test directory... 2019-10-17T20:36:25.997 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:25.998 INFO:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2019-10-17T20:36:26.003 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:26.003 INFO:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2019-10-17T20:36:26.010 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-10-17T20:36:26.042 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-10-17T20:36:26.042 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:26.042 INFO:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2019-10-17T20:36:26.051 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:36:26.052 INFO:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2019-10-17T20:36:26.064 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-10-17T20:36:26.102 INFO:teuthology.run_tasks:Running task kernel... 2019-10-17T20:36:26.120 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}} 2019-10-17T20:36:26.120 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-10-17T20:36:26.121 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:26.121 INFO:teuthology.orchestra.run.smithi092:> uname -r 2019-10-17T20:36:26.128 INFO:teuthology.orchestra.run.smithi092.stdout:4.15.0-58-generic 2019-10-17T20:36:26.128 INFO:teuthology.task.kernel:Running kernel on smithi092: 4.15.0-58-generic 2019-10-17T20:36:26.129 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:26.129 INFO:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2019-10-17T20:36:26.244 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:26.244 INFO:teuthology.orchestra.run.smithi092:> sudo apt-get update 2019-10-17T20:36:26.502 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-17T20:36:26.503 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-17T20:36:26.646 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-17T20:36:27.001 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [530 kB] 2019-10-17T20:36:27.037 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-17T20:36:27.398 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [753 kB] 2019-10-17T20:36:27.608 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [382 kB] 2019-10-17T20:36:28.040 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [593 kB] 2019-10-17T20:36:28.149 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [178 kB] 2019-10-17T20:36:28.623 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8,872 B] 2019-10-17T20:36:28.651 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [271 kB] 2019-10-17T20:36:28.785 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [3,296 B] 2019-10-17T20:36:28.946 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB] 2019-10-17T20:36:29.160 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [15.7 kB] 2019-10-17T20:36:29.401 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [4,956 B] 2019-10-17T20:36:29.563 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB] 2019-10-17T20:36:29.572 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB] 2019-10-17T20:36:30.184 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [204 kB] 2019-10-17T20:36:30.232 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB] 2019-10-17T20:36:30.666 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5,260 B] 2019-10-17T20:36:30.827 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2019-10-17T20:36:30.898 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [312 kB] 2019-10-17T20:36:30.989 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,464 B] 2019-10-17T20:36:31.440 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7,884 B] 2019-10-17T20:36:31.601 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,932 B] 2019-10-17T20:36:31.763 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3,944 B] 2019-10-17T20:36:31.924 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,020 B] 2019-10-17T20:36:32.087 INFO:teuthology.orchestra.run.smithi092.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,020 B] 2019-10-17T20:36:32.248 INFO:teuthology.orchestra.run.smithi092.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,856 B] 2019-10-17T20:36:32.370 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 6,748 kB in 6s (1,139 kB/s) 2019-10-17T20:36:33.654 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:36:33.697 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:33.698 INFO:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-17T20:36:33.786 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:36:34.028 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2019-10-17T20:36:34.028 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2019-10-17T20:36:34.099 INFO:teuthology.orchestra.run.smithi092.stderr:E: Unable to locate package linux-image-current-generic 2019-10-17T20:36:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-17T20:36:34.101 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:36:34.101 INFO:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-17T20:36:34.194 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:36:34.461 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2019-10-17T20:36:34.462 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2019-10-17T20:36:34.615 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2019-10-17T20:36:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: linux-generic linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-17T20:36:34.615 INFO:teuthology.orchestra.run.smithi092.stdout: linux-headers-generic linux-image-4.15.0-65-generic 2019-10-17T20:36:34.616 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-4.15.0-65-generic linux-modules-extra-4.15.0-65-generic 2019-10-17T20:36:34.617 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2019-10-17T20:36:34.617 INFO:teuthology.orchestra.run.smithi092.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-10-17T20:36:34.617 INFO:teuthology.orchestra.run.smithi092.stdout:Recommended packages: 2019-10-17T20:36:34.618 INFO:teuthology.orchestra.run.smithi092.stdout: thermald 2019-10-17T20:36:34.650 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2019-10-17T20:36:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-17T20:36:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: linux-image-4.15.0-65-generic linux-modules-4.15.0-65-generic 2019-10-17T20:36:34.651 INFO:teuthology.orchestra.run.smithi092.stdout: linux-modules-extra-4.15.0-65-generic 2019-10-17T20:36:34.652 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2019-10-17T20:36:34.653 INFO:teuthology.orchestra.run.smithi092.stdout: linux-generic linux-headers-generic linux-image-generic 2019-10-17T20:36:34.833 INFO:teuthology.orchestra.run.smithi092.stdout:3 upgraded, 5 newly installed, 0 to remove and 101 not upgraded. 2019-10-17T20:36:34.833 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 66.3 MB of archives. 2019-10-17T20:36:34.834 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 332 MB of additional disk space will be used. 2019-10-17T20:36:34.834 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-65-generic amd64 4.15.0-65.74 [13.1 MB] 2019-10-17T20:36:35.923 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-65-generic amd64 4.15.0-65.74 [7,986 kB] 2019-10-17T20:36:36.869 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-65-generic amd64 4.15.0-65.74 [32.9 MB] 2019-10-17T20:36:38.506 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.65.67 [1,868 B] 2019-10-17T20:36:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.65.67 [2,388 B] 2019-10-17T20:36:38.837 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65 all 4.15.0-65.74 [11.1 MB] 2019-10-17T20:36:39.876 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65-generic amd64 4.15.0-65.74 [1,237 kB] 2019-10-17T20:36:40.582 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.65.67 [2,348 B] 2019-10-17T20:36:40.987 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 66.3 MB in 6s (11.2 MB/s) 2019-10-17T20:36:41.150 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-4.15.0-65-generic. 2019-10-17T20:36:41.186 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101287 files and directories currently installed.) 2019-10-17T20:36:41.188 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../0-linux-modules-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:36:41.239 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:42.965 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-image-4.15.0-65-generic. 2019-10-17T20:36:42.978 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../1-linux-image-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:36:43.077 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:43.560 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-65-generic. 2019-10-17T20:36:43.565 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:36:43.601 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:47.816 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../3-linux-generic_4.15.0.65.67_amd64.deb ... 2019-10-17T20:36:47.939 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-17T20:36:48.271 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.65.67_amd64.deb ... 2019-10-17T20:36:48.392 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-image-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-17T20:36:48.720 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-headers-4.15.0-65. 2019-10-17T20:36:48.732 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../5-linux-headers-4.15.0-65_4.15.0-65.74_all.deb ... 2019-10-17T20:36:48.769 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-17T20:36:53.449 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package linux-headers-4.15.0-65-generic. 2019-10-17T20:36:53.462 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../6-linux-headers-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:36:53.499 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:55.871 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.65.67_amd64.deb ... 2019-10-17T20:36:56.019 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking linux-headers-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-17T20:36:56.426 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-17T20:36:56.540 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:57.033 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:57.168 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-headers-generic (4.15.0.65.67) ... 2019-10-17T20:36:57.294 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:57.783 INFO:teuthology.orchestra.run.smithi092.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-65-generic 2019-10-17T20:36:57.784 INFO:teuthology.orchestra.run.smithi092.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-65-generic 2019-10-17T20:36:57.921 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:58.414 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-image-generic (4.15.0.65.67) ... 2019-10-17T20:36:58.523 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up linux-generic (4.15.0.65.67) ... 2019-10-17T20:36:58.633 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:36:58.829 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-10-17T20:36:58.829 INFO:teuthology.orchestra.run.smithi092.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-65-generic 2019-10-17T20:37:11.748 INFO:teuthology.orchestra.run.smithi092.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-10-17T20:37:11.748 INFO:teuthology.orchestra.run.smithi092.stdout:Sourcing file `/etc/default/grub' 2019-10-17T20:37:11.767 INFO:teuthology.orchestra.run.smithi092.stdout:Generating grub configuration file ... 2019-10-17T20:37:12.077 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-17T20:37:12.101 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-17T20:37:12.378 INFO:teuthology.orchestra.run.smithi092.stdout:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-17T20:37:12.407 INFO:teuthology.orchestra.run.smithi092.stdout:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-17T20:37:13.256 INFO:teuthology.orchestra.run.smithi092.stdout:done 2019-10-17T20:37:13.644 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:37:13.644 INFO:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic 2019-10-17T20:37:13.669 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic 2019-10-17T20:37:13.669 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2019-10-17T20:37:13.669 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2019-10-17T20:37:13.669 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2019-10-17T20:37:13.669 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 15 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 4.15.0.65.67 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2019-10-17T20:37:13.670 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2019-10-17T20:37:13.671 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2019-10-17T20:37:13.671 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2019-10-17T20:37:13.671 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-58-generic Expected: 4.15.0-65-generic 2019-10-17T20:37:13.671 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:37:13.671 INFO:teuthology.orchestra.run.smithi204:> uname -r 2019-10-17T20:37:13.679 INFO:teuthology.orchestra.run.smithi204.stdout:4.15.0-58-generic 2019-10-17T20:37:13.679 INFO:teuthology.task.kernel:Running kernel on smithi204: 4.15.0-58-generic 2019-10-17T20:37:13.679 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:37:13.679 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2019-10-17T20:37:14.108 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:37:14.109 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get update 2019-10-17T20:37:14.383 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-17T20:37:14.384 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-17T20:37:14.530 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-17T20:37:14.843 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [382 kB] 2019-10-17T20:37:14.942 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-17T20:37:15.344 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [593 kB] 2019-10-17T20:37:15.412 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [530 kB] 2019-10-17T20:37:15.959 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [753 kB] 2019-10-17T20:37:16.006 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [178 kB] 2019-10-17T20:37:16.455 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8,872 B] 2019-10-17T20:37:16.602 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [271 kB] 2019-10-17T20:37:16.617 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [3,296 B] 2019-10-17T20:37:16.778 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [613 kB] 2019-10-17T20:37:17.128 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [15.7 kB] 2019-10-17T20:37:17.377 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [4,956 B] 2019-10-17T20:37:17.396 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [593 kB] 2019-10-17T20:37:17.559 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,012 kB] 2019-10-17T20:37:18.000 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [204 kB] 2019-10-17T20:37:18.254 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [982 kB] 2019-10-17T20:37:18.487 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2019-10-17T20:37:18.655 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5,260 B] 2019-10-17T20:37:18.817 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,464 B] 2019-10-17T20:37:18.927 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [312 kB] 2019-10-17T20:37:19.476 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7,884 B] 2019-10-17T20:37:19.664 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [6,932 B] 2019-10-17T20:37:19.849 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [3,944 B] 2019-10-17T20:37:20.014 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,020 B] 2019-10-17T20:37:20.183 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,020 B] 2019-10-17T20:37:20.355 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,856 B] 2019-10-17T20:37:20.494 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 6,748 kB in 6s (1,094 kB/s) 2019-10-17T20:37:21.831 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:37:21.876 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:37:21.877 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-17T20:37:21.959 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:37:22.207 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-17T20:37:22.207 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-17T20:37:22.287 INFO:teuthology.orchestra.run.smithi204.stderr:E: Unable to locate package linux-image-current-generic 2019-10-17T20:37:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-17T20:37:22.290 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:37:22.291 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-17T20:37:22.372 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:37:22.591 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-17T20:37:22.592 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-17T20:37:22.747 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2019-10-17T20:37:22.748 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-17T20:37:22.748 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-generic linux-image-4.15.0-65-generic 2019-10-17T20:37:22.748 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-4.15.0-65-generic linux-modules-extra-4.15.0-65-generic 2019-10-17T20:37:22.749 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2019-10-17T20:37:22.750 INFO:teuthology.orchestra.run.smithi204.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2019-10-17T20:37:22.750 INFO:teuthology.orchestra.run.smithi204.stdout:Recommended packages: 2019-10-17T20:37:22.750 INFO:teuthology.orchestra.run.smithi204.stdout: thermald 2019-10-17T20:37:22.782 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2019-10-17T20:37:22.782 INFO:teuthology.orchestra.run.smithi204.stdout: linux-headers-4.15.0-65 linux-headers-4.15.0-65-generic 2019-10-17T20:37:22.782 INFO:teuthology.orchestra.run.smithi204.stdout: linux-image-4.15.0-65-generic linux-modules-4.15.0-65-generic 2019-10-17T20:37:22.782 INFO:teuthology.orchestra.run.smithi204.stdout: linux-modules-extra-4.15.0-65-generic 2019-10-17T20:37:22.784 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2019-10-17T20:37:22.784 INFO:teuthology.orchestra.run.smithi204.stdout: linux-generic linux-headers-generic linux-image-generic 2019-10-17T20:37:22.963 INFO:teuthology.orchestra.run.smithi204.stdout:3 upgraded, 5 newly installed, 0 to remove and 101 not upgraded. 2019-10-17T20:37:22.963 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 66.3 MB of archives. 2019-10-17T20:37:22.964 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 332 MB of additional disk space will be used. 2019-10-17T20:37:22.964 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-65-generic amd64 4.15.0-65.74 [13.1 MB] 2019-10-17T20:37:24.029 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-65-generic amd64 4.15.0-65.74 [7,986 kB] 2019-10-17T20:37:24.981 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-65-generic amd64 4.15.0-65.74 [32.9 MB] 2019-10-17T20:37:26.596 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.65.67 [1,868 B] 2019-10-17T20:37:26.757 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.65.67 [2,388 B] 2019-10-17T20:37:26.919 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65 all 4.15.0-65.74 [11.1 MB] 2019-10-17T20:37:27.942 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-65-generic amd64 4.15.0-65.74 [1,237 kB] 2019-10-17T20:37:28.638 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.65.67 [2,348 B] 2019-10-17T20:37:29.128 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 66.3 MB in 6s (11.4 MB/s) 2019-10-17T20:37:29.281 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-4.15.0-65-generic. 2019-10-17T20:37:29.318 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101287 files and directories currently installed.) 2019-10-17T20:37:29.320 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../0-linux-modules-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:37:29.362 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:31.106 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-image-4.15.0-65-generic. 2019-10-17T20:37:31.118 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../1-linux-image-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:37:31.241 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:31.743 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-65-generic. 2019-10-17T20:37:31.748 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:37:31.777 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:35.742 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../3-linux-generic_4.15.0.65.67_amd64.deb ... 2019-10-17T20:37:35.863 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-17T20:37:36.196 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.65.67_amd64.deb ... 2019-10-17T20:37:36.300 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-image-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-17T20:37:36.653 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-4.15.0-65. 2019-10-17T20:37:36.665 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../5-linux-headers-4.15.0-65_4.15.0-65.74_all.deb ... 2019-10-17T20:37:36.703 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-17T20:37:41.383 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package linux-headers-4.15.0-65-generic. 2019-10-17T20:37:41.402 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../6-linux-headers-4.15.0-65-generic_4.15.0-65.74_amd64.deb ... 2019-10-17T20:37:41.458 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:43.880 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.65.67_amd64.deb ... 2019-10-17T20:37:44.003 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking linux-headers-generic (4.15.0.65.67) over (4.15.0.58.60) ... 2019-10-17T20:37:44.315 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-4.15.0-65 (4.15.0-65.74) ... 2019-10-17T20:37:44.441 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:45.635 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:45.752 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-headers-generic (4.15.0.65.67) ... 2019-10-17T20:37:45.870 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:46.295 INFO:teuthology.orchestra.run.smithi204.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-65-generic 2019-10-17T20:37:46.295 INFO:teuthology.orchestra.run.smithi204.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-65-generic 2019-10-17T20:37:46.431 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-modules-extra-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:46.924 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-image-generic (4.15.0.65.67) ... 2019-10-17T20:37:47.050 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up linux-generic (4.15.0.65.67) ... 2019-10-17T20:37:47.176 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for linux-image-4.15.0-65-generic (4.15.0-65.74) ... 2019-10-17T20:37:47.357 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/initramfs-tools: 2019-10-17T20:37:47.357 INFO:teuthology.orchestra.run.smithi204.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-65-generic 2019-10-17T20:38:00.461 INFO:teuthology.orchestra.run.smithi204.stdout:/etc/kernel/postinst.d/zz-update-grub: 2019-10-17T20:38:00.461 INFO:teuthology.orchestra.run.smithi204.stdout:Sourcing file `/etc/default/grub' 2019-10-17T20:38:00.496 INFO:teuthology.orchestra.run.smithi204.stdout:Generating grub configuration file ... 2019-10-17T20:38:00.823 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-17T20:38:00.855 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-17T20:38:01.157 INFO:teuthology.orchestra.run.smithi204.stdout:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-17T20:38:01.184 INFO:teuthology.orchestra.run.smithi204.stdout:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-17T20:38:02.018 INFO:teuthology.orchestra.run.smithi204.stdout:done 2019-10-17T20:38:02.395 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:02.396 INFO:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2019-10-17T20:38:02.419 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2019-10-17T20:38:02.419 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2019-10-17T20:38:02.420 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2019-10-17T20:38:02.420 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 15 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 4.15.0.65.67 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-17T20:38:02.421 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2019-10-17T20:38:02.422 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2019-10-17T20:38:02.422 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2019-10-17T20:38:02.422 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2019-10-17T20:38:02.422 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-58-generic Expected: 4.15.0-65-generic 2019-10-17T20:38:02.422 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-10-17T20:38:02.422 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-17T20:38:02.422 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-10-17T20:38:02.423 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-10-17T20:38:02.423 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2019-10-17T20:38:02.423 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:02.423 INFO:teuthology.orchestra.run.smithi092:> sudo apt-get clean 2019-10-17T20:38:02.472 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:02.472 INFO:teuthology.orchestra.run.smithi092:> sudo apt-get update 2019-10-17T20:38:02.733 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-17T20:38:02.740 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-17T20:38:02.882 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-17T20:38:03.043 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-17T20:38:04.432 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:38:04.478 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:04.478 INFO:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-17T20:38:04.557 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:38:04.780 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2019-10-17T20:38:04.780 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2019-10-17T20:38:04.851 INFO:teuthology.orchestra.run.smithi092.stderr:E: Unable to locate package linux-image-current-generic 2019-10-17T20:38:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-17T20:38:04.853 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:04.854 INFO:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-17T20:38:04.931 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:38:05.179 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2019-10-17T20:38:05.179 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2019-10-17T20:38:05.368 INFO:teuthology.orchestra.run.smithi092.stdout:linux-image-generic is already the newest version (4.15.0.65.67). 2019-10-17T20:38:05.368 INFO:teuthology.orchestra.run.smithi092.stdout:0 upgraded, 0 newly installed, 0 to remove and 101 not upgraded. 2019-10-17T20:38:05.370 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.370 INFO:teuthology.orchestra.run.smithi092:> dpkg -s linux-image-generic 2019-10-17T20:38:05.392 INFO:teuthology.orchestra.run.smithi092.stdout:Package: linux-image-generic 2019-10-17T20:38:05.392 INFO:teuthology.orchestra.run.smithi092.stdout:Status: install ok installed 2019-10-17T20:38:05.392 INFO:teuthology.orchestra.run.smithi092.stdout:Priority: optional 2019-10-17T20:38:05.392 INFO:teuthology.orchestra.run.smithi092.stdout:Section: kernel 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Installed-Size: 15 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Maintainer: Ubuntu Kernel Team 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: amd64 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Source: linux-meta 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Version: 4.15.0.65.67 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-17T20:38:05.393 INFO:teuthology.orchestra.run.smithi092.stdout:Recommends: thermald 2019-10-17T20:38:05.394 INFO:teuthology.orchestra.run.smithi092.stdout:Description: Generic Linux kernel image 2019-10-17T20:38:05.394 INFO:teuthology.orchestra.run.smithi092.stdout: This package will always depend on the latest generic kernel image 2019-10-17T20:38:05.394 INFO:teuthology.orchestra.run.smithi092.stdout: available. 2019-10-17T20:38:05.394 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.394 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:38:05.465 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.i9gi7OBelE 2019-10-17T20:38:05.465 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.465 INFO:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub/grub.cfg /tmp/tmp.i9gi7OBelE 2019-10-17T20:38:05.548 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.549 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.i9gi7OBelE 2019-10-17T20:38:05.717 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.i9gi7OBelE is 8KB 2019-10-17T20:38:05.771 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.771 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.i9gi7OBelE 2019-10-17T20:38:05.782 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.782 INFO:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-10-17T20:38:05.862 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.862 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-10-17T20:38:05.983 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-65-generic 2019-10-17T20:38:05.984 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:05.984 INFO:teuthology.orchestra.run.smithi092:> sudo update-grub 2019-10-17T20:38:07.138 INFO:teuthology.orchestra.run.smithi092.stderr:Sourcing file `/etc/default/grub' 2019-10-17T20:38:07.164 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2019-10-17T20:38:07.403 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-17T20:38:07.415 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-17T20:38:07.677 INFO:teuthology.orchestra.run.smithi092.stderr:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-17T20:38:07.688 INFO:teuthology.orchestra.run.smithi092.stderr:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-17T20:38:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:done 2019-10-17T20:38:08.220 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:38:08.220 INFO:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2019-10-17T20:38:08.224 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-10-17T20:38:08.224 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2019-10-17T20:38:08.224 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:08.224 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get clean 2019-10-17T20:38:08.285 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:08.285 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get update 2019-10-17T20:38:08.407 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2019-10-17T20:38:08.524 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-17T20:38:08.682 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2019-10-17T20:38:08.843 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2019-10-17T20:38:10.200 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:38:10.249 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:10.250 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2019-10-17T20:38:10.343 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:38:10.573 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-17T20:38:10.573 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-17T20:38:10.650 INFO:teuthology.orchestra.run.smithi204.stderr:E: Unable to locate package linux-image-current-generic 2019-10-17T20:38:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 100 2019-10-17T20:38:10.652 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:10.652 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2019-10-17T20:38:10.726 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:38:10.945 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-17T20:38:10.946 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-17T20:38:11.124 INFO:teuthology.orchestra.run.smithi204.stdout:linux-image-generic is already the newest version (4.15.0.65.67). 2019-10-17T20:38:11.124 INFO:teuthology.orchestra.run.smithi204.stdout:0 upgraded, 0 newly installed, 0 to remove and 101 not upgraded. 2019-10-17T20:38:11.125 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.126 INFO:teuthology.orchestra.run.smithi204:> dpkg -s linux-image-generic 2019-10-17T20:38:11.149 INFO:teuthology.orchestra.run.smithi204.stdout:Package: linux-image-generic 2019-10-17T20:38:11.149 INFO:teuthology.orchestra.run.smithi204.stdout:Status: install ok installed 2019-10-17T20:38:11.149 INFO:teuthology.orchestra.run.smithi204.stdout:Priority: optional 2019-10-17T20:38:11.149 INFO:teuthology.orchestra.run.smithi204.stdout:Section: kernel 2019-10-17T20:38:11.149 INFO:teuthology.orchestra.run.smithi204.stdout:Installed-Size: 15 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Maintainer: Ubuntu Kernel Team 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: amd64 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Source: linux-meta 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Version: 4.15.0.65.67 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Depends: linux-image-4.15.0-65-generic, linux-modules-extra-4.15.0-65-generic, linux-firmware, intel-microcode, amd64-microcode 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Recommends: thermald 2019-10-17T20:38:11.150 INFO:teuthology.orchestra.run.smithi204.stdout:Description: Generic Linux kernel image 2019-10-17T20:38:11.151 INFO:teuthology.orchestra.run.smithi204.stdout: This package will always depend on the latest generic kernel image 2019-10-17T20:38:11.151 INFO:teuthology.orchestra.run.smithi204.stdout: available. 2019-10-17T20:38:11.151 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.151 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:38:11.225 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.5p15R1N7oz 2019-10-17T20:38:11.226 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.226 INFO:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5p15R1N7oz 2019-10-17T20:38:11.298 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.298 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.5p15R1N7oz 2019-10-17T20:38:11.467 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.5p15R1N7oz is 8KB 2019-10-17T20:38:11.522 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.523 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.5p15R1N7oz 2019-10-17T20:38:11.531 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.532 INFO:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2019-10-17T20:38:11.612 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.612 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2019-10-17T20:38:11.731 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-65-generic 2019-10-17T20:38:11.731 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:11.731 INFO:teuthology.orchestra.run.smithi204:> sudo update-grub 2019-10-17T20:38:13.020 INFO:teuthology.orchestra.run.smithi204.stderr:Sourcing file `/etc/default/grub' 2019-10-17T20:38:13.037 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2019-10-17T20:38:13.268 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-4.15.0-65-generic 2019-10-17T20:38:13.279 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-4.15.0-65-generic 2019-10-17T20:38:13.557 INFO:teuthology.orchestra.run.smithi204.stderr:Found linux image: /boot/vmlinuz-4.15.0-58-generic 2019-10-17T20:38:13.567 INFO:teuthology.orchestra.run.smithi204.stderr:Found initrd image: /boot/initrd.img-4.15.0-58-generic 2019-10-17T20:38:14.030 INFO:teuthology.orchestra.run.smithi204.stderr:done 2019-10-17T20:38:14.032 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:38:14.033 INFO:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2019-10-17T20:38:14.036 INFO:teuthology.misc:Re-opening connections... 2019-10-17T20:38:14.036 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:38:14.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:38:14.037 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:39:14.039 DEBUG:teuthology.orchestra.remote:timed out 2019-10-17T20:39:14.039 INFO:teuthology.misc:trying to connect to ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:39:14.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:39:14.039 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:39:48.668 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2019-10-17T20:39:48.669 DEBUG:teuthology.misc:waited 94.6322309971 2019-10-17T20:39:49.670 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:39:49.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:39:49.670 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:39:52.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2019-10-17T20:39:52.729 INFO:teuthology.misc:trying to connect to ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:39:52.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:39:52.729 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:39:55.800 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2019-10-17T20:39:55.801 DEBUG:teuthology.misc:waited 101.764219999 2019-10-17T20:39:56.801 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:39:56.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:39:56.802 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:39:59.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2019-10-17T20:39:59.869 INFO:teuthology.misc:trying to connect to ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:39:59.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:39:59.870 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:39:59.945 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:39:59.945 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-17T20:40:00.215 DEBUG:teuthology.misc:waited 106.178736925 2019-10-17T20:40:01.216 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:40:01.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:40:01.217 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:40:02.936 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2019-10-17T20:40:02.937 DEBUG:teuthology.misc:waited 108.900382042 2019-10-17T20:40:03.938 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:40:03.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:40:03.939 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:40:03.961 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2019-10-17T20:40:03.961 DEBUG:teuthology.misc:waited 109.924421072 2019-10-17T20:40:04.962 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:40:04.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:40:04.963 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:40:04.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2019-10-17T20:40:04.964 DEBUG:teuthology.misc:waited 110.927464008 2019-10-17T20:40:05.964 INFO:teuthology.misc:trying to connect to ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:40:05.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:40:05.965 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:40:06.099 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:06.100 INFO:teuthology.orchestra.run.smithi092:> true 2019-10-17T20:40:06.405 DEBUG:teuthology.misc:waited 112.368921995 2019-10-17T20:40:07.406 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-10-17T20:40:07.406 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-65-generic"... 2019-10-17T20:40:07.407 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:07.407 INFO:teuthology.orchestra.run.smithi092:> uname -r 2019-10-17T20:40:07.414 INFO:teuthology.orchestra.run.smithi092.stdout:4.15.0-65-generic 2019-10-17T20:40:07.414 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-65-generic 2019-10-17T20:40:07.414 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-17T20:40:07.415 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-10-17T20:40:07.415 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-65-generic"... 2019-10-17T20:40:07.415 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:07.415 INFO:teuthology.orchestra.run.smithi204:> uname -r 2019-10-17T20:40:07.425 INFO:teuthology.orchestra.run.smithi204.stdout:4.15.0-65-generic 2019-10-17T20:40:07.425 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-65-generic 2019-10-17T20:40:07.425 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-10-17T20:40:08.426 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-10-17T20:40:08.426 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.426 INFO:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-17T20:40:08.458 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2019-10-17T20:40:08.493 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-10-17T20:40:08.493 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:08.493 INFO:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-10-17T20:40:08.525 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2019-10-17T20:40:08.544 INFO:teuthology.run_tasks:Running task internal.base... 2019-10-17T20:40:08.553 INFO:teuthology.task.internal:Creating test directory... 2019-10-17T20:40:08.553 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.553 INFO:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-17T20:40:08.559 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:08.559 INFO:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-10-17T20:40:08.599 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-10-17T20:40:08.606 INFO:teuthology.run_tasks:Running task internal.archive... 2019-10-17T20:40:08.615 INFO:teuthology.task.internal:Creating archive directory... 2019-10-17T20:40:08.615 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.615 INFO:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-17T20:40:08.621 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:08.621 INFO:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-10-17T20:40:08.667 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-10-17T20:40:08.675 INFO:teuthology.task.internal:Enabling coredump saving... 2019-10-17T20:40:08.675 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.675 INFO:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-17T20:40:08.679 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:08.679 INFO:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-17T20:40:08.694 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-17T20:40:08.737 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-10-17T20:40:08.738 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-10-17T20:40:08.745 INFO:teuthology.task.internal:Configuring sudo... 2019-10-17T20:40:08.746 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.746 INFO:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-17T20:40:08.751 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:08.751 INFO:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-10-17T20:40:08.800 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-10-17T20:40:08.810 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-10-17T20:40:08.810 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.811 INFO:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-17T20:40:08.814 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:08.814 INFO:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-10-17T20:40:08.854 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.854 INFO:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-17T20:40:08.870 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.872 INFO:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-17T20:40:08.943 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:08.944 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-17T20:40:09.057 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:09.057 INFO:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-10-17T20:40:09.065 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:09.065 INFO:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-10-17T20:40:09.137 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:09.137 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-10-17T20:40:09.250 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:40:09.250 INFO:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2019-10-17T20:40:09.254 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:40:09.254 INFO:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2019-10-17T20:40:09.355 INFO:teuthology.run_tasks:Running task internal.timer... 2019-10-17T20:40:09.365 INFO:teuthology.task.internal:Starting timer... 2019-10-17T20:40:09.365 INFO:teuthology.run_tasks:Running task pcp... 2019-10-17T20:40:09.401 INFO:teuthology.run_tasks:Running task selinux... 2019-10-17T20:40:09.419 DEBUG:teuthology.task.selinux:Excluding smithi092: OS 'ubuntu' does not support SELinux 2019-10-17T20:40:09.419 DEBUG:teuthology.task.selinux:Excluding smithi204: OS 'ubuntu' does not support SELinux 2019-10-17T20:40:09.419 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-10-17T20:40:09.419 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2019-10-17T20:40:09.420 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-10-17T20:40:09.420 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-10-17T20:40:09.431 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2019-10-17T20:40:09.460 INFO:teuthology.repo_utils:Fetching master from upstream 2019-10-17T20:40:09.547 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch master 2019-10-17T20:40:09.583 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'set_python_path.yml'}, {'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-10-17T20:40:09.586 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-10-17T20:40:09.994 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-10-17T20:40:13.602 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-17T20:40:13.621 INFO:teuthology.task.ansible.out: TASK [Check for /usr/bin/python] *********************************************** 2019-10-17T20:40:13.622 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:13 +0000 (0:00:00.058) 0:00:00.058 ****** 2019-10-17T20:40:15.281 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi092.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:40:15.282 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.003175", "end": "2019-10-17 20:40:15.255115", "rc": 0, "start": "2019-10-17 20:40:15.251940", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-17T20:40:15.586 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi204.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:40:15.588 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo marco", "delta": "0:00:00.003224", "end": "2019-10-17 20:40:15.775643", "rc": 0, "start": "2019-10-17 20:40:15.772419", "stderr": "", "stderr_lines": [], "stdout": "marco", "stdout_lines": ["marco"]} 2019-10-17T20:40:15.594 INFO:teuthology.task.ansible.out: TASK [Set ansible_python_interpreter=/usr/bin/python3] ************************* 2019-10-17T20:40:15.596 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:15 +0000 (0:00:01.973) 0:00:02.032 ****** 2019-10-17T20:40:15.621 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:15.631 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:15.638 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-10-17T20:40:15.648 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:15 +0000 (0:00:00.052) 0:00:02.085 ****** 2019-10-17T20:40:15.662 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:15 +0000 (0:00:00.014) 0:00:02.100 ****** 2019-10-17T20:40:16.813 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.204"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:77ab"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "08/02/2016", "ansible_bios_version": "2.0a", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-17", "day": "17", "epoch": "1571344816", "hour": "20", "iso8601": "2019-10-17T20:40:16Z", "iso8601_basic": "20191017T204016936231", "iso8601_basic_short": "20191017T204016", "iso8601_micro": "2019-10-17T20:40:16.936319Z", "minute": "40", "month": "10", "second": "16", "time": "20:40:16", "tz": "UTC", "tz_offset": "+0000", "weekday": "Thursday", "weekday_number": "4", "weeknumber": "41", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.204", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:77ab", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjV 2019-10-17T20:40:16.815 INFO:teuthology.task.ansible.out:KWIPr7i7oCSEddf"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB"], "nvme0n1": ["lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["56e60058-f0cb-47e1-a3d6-bf071031bc28"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi"], "labels": [], "masters": [], "uuids": ["56e60058-f0cb-47e1-a3d6-bf071031bc28"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": nul 2019-10-17T20:40:16.816 INFO:teuthology.task.ansible.out:l, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_han 2019-10-17T20:40:16.818 INFO:teuthology.task.ansible.out:dle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1 2019-10-17T20:40:16.819 INFO:teuthology.task.ansible.out:", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941a36"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], 2019-10-17T20:40:16.820 INFO:teuthology.task.ansible.out: "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7 2019-10-17T20:40:16.822 INFO:teuthology.task.ansible.out:a:d9:36:2a", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx 2019-10-17T20:40:16.823 INFO:teuthology.task.ansible.out:_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2b", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx 2019-10-17T20:40:16.825 INFO:teuthology.task.ansible.out:_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:77:aa", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", 2019-10-17T20:40:16.826 INFO:teuthology.task.ansible.out:"tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.204", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:77ab", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:77:ab", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-jibdkrevvalttjrspxcpkbaueupcbqte ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi204.front.sepia.ceph.com", "ansible_hostname": "smithi204", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "lo", "enp3s0f0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-ge 2019-10-17T20:40:16.827 INFO:teuthology.task.ansible.out:neric", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0" 2019-10-17T20:40:16.829 INFO:teuthology.task.ansible.out:}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31475, "ansible_memory_mb": {"nocache": {"free": 31783, "used": 271}, "real": {"free": 31475, "total": 32054, "used": 579}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32054, "ansible_mounts": [{"block_available": 226440791, "block_size": 4096, "block_total": 240095108, "block_used": 13654317, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501479936, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi204.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Int 2019-10-17T20:40:16.830 INFO:teuthology.task.ansible.out:el(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6928378", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9362A", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANMEck9GESxvxmlPQWWAc9+xWWX9ojaYleSG83jVU7pbOPa5kU4W3YyH//P/AG992WM8UqJZYn2ZLg0C58mCq0fSRfC5wJ6KHs3l+F/tEhBAekwPJnVYvAn5IKR/+cnepj+PWg2tlh2AU3Svnnlhvs8bNcinr+AA7YguVJRvK5oZAAAAFQDs4EBMn+x6HWk4zzRCO5PeGHusEwAAAIEAn1HXkqBFNVykrt51uPPrjrp2vWX3sD5PnWMvvG9Qovl0ufs9BypZEtQWw2yqac4EquTMltJUJpA+Vz5ilOeS/UlYsaQ2OQo6GokoOS5WBKphA3dnfSOWO4Wpewpp7T2eLZ553KXhtFz3JI25/ujPMu+beKexotnpXtcMQ/qqqNQAAACAAd2EY3J5qqF35pLkfFljo0akGaZZiJpm9Jt7fCpOa5IGKPFQuM0+vvYmE2HlAl6oC+/LsKdZHqiS8h98caZWgSKL3VvT211qCCLlPUJvXevF/MRac7A3Ja6mCU+Mn4/CPxJZgaCRXsbxFpmC7OFZFqzeQYrDwJGQ5I22KAX83PQ=", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqsZalpKBKZAqxYwnXIF2yLGlgp/Xb3zi+PPJP9/u1OM7A9JToGvk7AJqt6m4SnGOK1WKXXgV7QUdikuQqZmH0=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1 2019-10-17T20:40:16.832 INFO:teuthology.task.ansible.out:lZDI1NTE5AAAAIF8H48K0MbVIg4Oyt7fLXDjrNI4nR94y//Z98RLeF6Om", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCxcgkWcgpyXJpHPdcMl8MDrjKlUgM3Us9wefFAGIALtC4e2S4bWyY/L9DXlacl7CJCIIhy6oB7yzN/KZ35HJskpcwp6GAWk0PMRHnhOTC5xioN5fskgms3DKh5UXWqb9Ulyn4UWNRkKv20J5Lr43Jfo0qZ9QlSLRVf 2019-10-17T20:40:16.841 INFO:teuthology.task.ansible.out:+IrVLturRG49z2bZYvi2yapEuI28/HezE9qpx5Hd1L5qWQ4Fmjpz3XVjrjMgDduhjCEbyuQj5xGp8xFREZB1fYTfJ6JocCgqCfxH58cWTEB+c2ZFxV5HFrN9/VWLTtWlVaS6N4qNNmy1uQh/qUA94Jqb02P8Svi2/7YwIsYBhixu6JCBvGu9", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 33, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-17T20:40:16.918 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:16 +0000 (0:00:01.255) 0:00:03.355 ****** 2019-10-17T20:40:16.938 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.92"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe8f:cd45"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-17", "day": "17", "epoch": "1571344816", "hour": "20", "iso8601": "2019-10-17T20:40:16Z", "iso8601_basic": "20191017T204016840299", "iso8601_basic_short": "20191017T204016", "iso8601_micro": "2019-10-17T20:40:16.840414Z", "minute": "40", "month": "10", "second": "16", "time": "20:40:16", "tz": "UTC", "tz_offset": "+0000", "weekday": "Thursday", "weekday_number": "4", "weeknumber": "41", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.92", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cd:45", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe8f:cd45", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cd:45", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0L 2019-10-17T20:40:16.939 INFO:teuthology.task.ansible.out:iEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR"], "nvme0n1": ["lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN", "nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S", "wwn-0x5000c500918da5aa"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1", "wwn-0x5000c500918da5aa-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["a1e335b0-c538-47e5-8baf-4d144ca2ebf1"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X"], "labels": [], "masters": [], "uuids": ["a1e335b0-c538-47e5-8baf-4d144ca2ebf1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB 2019-10-17T20:40:16.941 INFO:teuthology.task.ansible.out:", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.0 2019-10-17T20:40:16.942 INFO:teuthology.task.ansible.out:0 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contro 2019-10-17T20:40:16.944 INFO:teuthology.task.ansible.out:ller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN", "nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S", "wwn-0x5000c500918da5aa"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1", "wwn-0x5000c500918da5aa-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJ4S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918da5aa"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "an 2019-10-17T20:40:16.945 INFO:teuthology.task.ansible.out:sible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]" 2019-10-17T20:40:16.946 INFO:teuthology.task.ansible.out:, "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:a1:b2", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp 2019-10-17T20:40:16.948 INFO:teuthology.task.ansible.out:_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:a1:b3", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "featur 2019-10-17T20:40:16.949 INFO:teuthology.task.ansible.out:es": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:8f:cd:44", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": 2019-10-17T20:40:16.951 INFO:teuthology.task.ansible.out:true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.92", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe8f:cd45", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:8f 2019-10-17T20:40:16.952 INFO:teuthology.task.ansible.out::cd:45", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-wbhdslkgyopymlxsfypmsuqmzfcsgawd ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi092.front.sepia.ceph.com", "ansible_hostname": "smithi092", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "enp3s0f0", "eno2", "lo", "eno1"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed 2019-10-17T20:40:16.954 INFO:teuthology.task.ansible.out:]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": 2019-10-17T20:40:16.955 INFO:teuthology.task.ansible.out: "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31468, "ansible_memory_mb": {"nocache": {"free": 31784, "used": 271}, "real": {"free": 31468, "total": 32055, "used": 587}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32055, "ansible_mounts": [{"block_available": 226436697, "block_size": 4096, "block_tot 2019-10-17T20:40:16.956 INFO:teuthology.task.ansible.out:al": 240095108, "block_used": 13658411, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906479, "inode_total": 61054976, "inode_used": 148497, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927484710912, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi092.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6731596", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD8A1B2", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_ds 2019-10-17T20:40:16.958 INFO:teuthology.task.ansible.out:a_public": "AAAAB3NzaC1kc3MAAACBAKk466I9Uf03hemUJNgcLFV4ZRWcNC/gRyBAmrtAr9I8j9iekUrtLn9VCkceY9pLBuMwAeJds9pazjq8P2G7CoxD5JGiz2yyPnbxw7tHAaVyWuwvkzKjxj0AsZ373Dulgr8ja9gtC8rK51OELoD+uRFpaYmnpd3BWPHSpfk5r94lAAAAFQCARnjyQQt8ppmJCDWWQW4Zn2iePwAAAIEAgQCZ7jGxDteo2c0uGiXvsv5u1/z+VE4k7/LSitNpnDhlRFfPhgphLxpY6E7WWqIR9BQtQ0l06xoiJu8V0wQAQoS6eZ2bJaZJ0Kf7anTpeh+3EsbIf0Pg/aAO7hnzjupEweT5lgdy1JPVF4ejsRztb6E+HpzQxKHYBrZu/2glGuIAAACBAJbFrS+a5ZRbMmizQAS1OmDZgs7wDf2mQoD20xql+Y4Um/n/a+87PVqqQz25ARCtXIzwJkCs62ZKJ/D9ST4iZwpJr2HWNWLML+TWBVmsuJoTHMrMEKmRAWIOC3EPMrJBhBlxfAMxIETb0/H4rZEhPW22njDDLnca7if1bIm9kLnB", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDpduaNKhllWNs6faZ50AY0cTA+YiCiArlQlGUbSv3W4knswJ3ZEgld8jAKCTOOMmsai22UeiJkKj57Xh5ZVdj0=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIJABoIeQJwUd7aDRWaC+IV2a6JyxfGl/fN40EIrgxTGT", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCvZvqdwAiqPH7Tv1SjnaVZbTB7C3O9xmcp8DXENfY2xFXll6gLqfr1/nmtdlJqRuSM68CE+wiFlpv/97UuOtY5a99TYfOzNqZL8qJxpfNs9wH12XDF2Jjyd9rLVwrH9pcARm4icefXOTSlntJt0YdOn/IJbjeeUulmAxLtMYzM/t4BD0TPrAi061MNPZamx99gWEZwZkN3FFTWHGNR+hKaZUqwSoFKGVxuSncL/Y/FJ27itleZRQTHTdPDbjeLe6e6yvgf/pLyAu/AvnIqrSyHBY+khT/2mYNqwfXY5mwZIBQRRGOI1RgRDxWuX7mp3oI5aCSE6BMFjw/NA5Dbv1RN", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", 2019-10-17T20:40:16.959 INFO:teuthology.task.ansible.out: "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 29, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-17T20:40:17.040 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:17 +0000 (0:00:00.122) 0:00:03.478 ****** 2019-10-17T20:40:17.170 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2019-10-17T20:40:17.171 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-17T20:40:17.172 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-17T20:40:17.211 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:17 +0000 (0:00:00.171) 0:00:03.649 ****** 2019-10-17T20:40:17.246 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:17 +0000 (0:00:00.034) 0:00:03.684 ****** 2019-10-17T20:40:17.674 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-10-17T20:40:17.675 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-17T20:40:17.719 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:17 +0000 (0:00:00.473) 0:00:04.157 ****** 2019-10-17T20:40:17.725 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-10-17T20:40:17.735 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:40:17.765 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:17 +0000 (0:00:00.045) 0:00:04.203 ****** 2019-10-17T20:40:17.789 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:40:17.914 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-10-17T20:40:17.915 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.058890", "end": "2019-10-17 20:40:17.885077", "rc": 0, "start": "2019-10-17 20:40:17.826187", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-10-17T20:40:17.960 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:17 +0000 (0:00:00.194) 0:00:04.397 ****** 2019-10-17T20:40:17.978 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.052611", "end": "2019-10-17 20:40:18.152247", "rc": 0, "start": "2019-10-17 20:40:18.099636", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2019-10-17T20:40:18.021 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:18 +0000 (0:00:00.060) 0:00:04.458 ****** 2019-10-17T20:40:18.222 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-10-17T20:40:18.223 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:40:18.225 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:40:18.265 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:18 +0000 (0:00:00.244) 0:00:04.703 ****** 2019-10-17T20:40:18.301 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:18 +0000 (0:00:00.035) 0:00:04.739 ****** 2019-10-17T20:40:19.065 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-10-17T20:40:19.066 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1571344818.28-228533465811531/source", "state": "file", "uid": 0} 2019-10-17T20:40:19.129 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:19 +0000 (0:00:00.827) 0:00:05.567 ****** 2019-10-17T20:40:19.138 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1571344818.32-221767274558713/source", "state": "file", "uid": 0} 2019-10-17T20:40:19.183 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:19 +0000 (0:00:00.053) 0:00:05.621 ****** 2019-10-17T20:40:19.445 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2019-10-17T20:40:19.445 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.457 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.582 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.593 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.707 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.718 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.832 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.855 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.959 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:19.992 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:20.086 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:20.138 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:20.201 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:20.294 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-10-17T20:40:20.402 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2019-10-17T20:40:20.403 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-10-17T20:40:20.412 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-10-17T20:40:20.617 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:20 +0000 (0:00:01.433) 0:00:07.054 ****** 2019-10-17T20:40:20.655 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:20 +0000 (0:00:00.036) 0:00:07.091 ****** 2019-10-17T20:40:21.312 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-10-17T20:40:21.322 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.92"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe8f:cd45"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "12/17/2015", "ansible_bios_version": "2.0", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-17", "day": "17", "epoch": "1571344821", "hour": "20", "iso8601": "2019-10-17T20:40:21Z", "iso8601_basic": "20191017T204021232954", "iso8601_basic_short": "20191017T204021", "iso8601_micro": "2019-10-17T20:40:21.233053Z", "minute": "40", "month": "10", "second": "21", "time": "20:40:21", "tz": "UTC", "tz_offset": "+0000", "weekday": "Thursday", "weekday_number": "4", "weeknumber": "41", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.92", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cd:45", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe8f:cd45", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:8f:cd:45", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0L 2019-10-17T20:40:21.323 INFO:teuthology.task.ansible.out:iEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR"], "nvme0n1": ["lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN", "nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S", "wwn-0x5000c500918da5aa"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1", "wwn-0x5000c500918da5aa-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["a1e335b0-c538-47e5-8baf-4d144ca2ebf1"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X"], "labels": [], "masters": [], "uuids": ["a1e335b0-c538-47e5-8baf-4d144ca2ebf1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB 2019-10-17T20:40:21.324 INFO:teuthology.task.ansible.out:", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.0 2019-10-17T20:40:21.326 INFO:teuthology.task.ansible.out:0 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory contro 2019-10-17T20:40:21.327 INFO:teuthology.task.ansible.out:ller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN", "nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S", "wwn-0x5000c500918da5aa"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1", "wwn-0x5000c500918da5aa-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJ4S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918da5aa"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, "an 2019-10-17T20:40:21.328 INFO:teuthology.task.ansible.out:sible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:a1:b2", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, "pr 2019-10-17T20:40:21.330 INFO:teuthology.task.ansible.out:omisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d8:a1:b3", "module": "igb", "mtu": 1500, "pciid": 2019-10-17T20:40:21.331 INFO:teuthology.task.ansible.out:"0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentatio 2019-10-17T20:40:21.333 INFO:teuthology.task.ansible.out:n": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:8f:cd:44", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmenta 2019-10-17T20:40:21.334 INFO:teuthology.task.ansible.out:tion": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.92", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe8f:cd45", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:8f:cd:45", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-svkkqvnbnlvihgvwheefmgrkejxwqgjb ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi092.front.sepia.ceph.com", "ansible_hostname": "smithi092", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "enp3s0f0", "eno2", "lo", "eno1"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-generic", "ansible_lo": {"active": true, "device": "lo 2019-10-17T20:40:21.335 INFO:teuthology.task.ansible.out:", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0"}, "ipv6": [{"address": "::1", "prefix": "128", "sco 2019-10-17T20:40:21.337 INFO:teuthology.task.ansible.out:pe": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31461, "ansible_memory_mb": {"nocache": {"free": 31783, "used": 272}, "real": {"free": 31461, "total": 32055, "used": 594}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32055, "ansible_mounts": [{"block_available": 226436689, "block_size": 4096, "block_total": 240095108, "block_used": 13658419, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906479, "inode_total": 61054976, "inode_used": 148497, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927484678144, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi092.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "Genui 2019-10-17T20:40:21.338 INFO:teuthology.task.ansible.out:neIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6731596", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD8A1B2", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBAKk466I9Uf03hemUJNgcLFV4ZRWcNC/gRyBAmrtAr9I8j9iekUrtLn9VCkceY9pLBuMwAeJds9pazjq8P2G7CoxD5JGiz2yyPnbxw7tHAaVyWuwvkzKjxj0AsZ373Dulgr8ja9gtC8rK51OELoD+uRFpaYmnpd3BWPHSpfk5r94lAAAAFQCARnjyQQt8ppmJCDWWQW4Zn2iePwAAAIEAgQCZ7jGxDteo2c0uGiXvsv5u1/z+VE4k7/LSitNpnDhlRFfPhgphLxpY6E7WWqIR9BQtQ0l06xoiJu8V0wQAQoS6eZ2bJaZJ0Kf7anTpeh+3EsbIf0Pg/aAO7hnzjupEweT5lgdy1JPVF4ejsRztb6E+HpzQxKHYBrZu/2glGuIAAACBAJbFrS+a5ZRbMmizQAS1OmDZgs7wDf2mQoD20xql+Y4Um/n/a+87PVqqQz25ARCtXIzwJkCs62ZKJ/D9ST4iZwpJr2HWNWLML+TWBVmsuJoTHMrMEKmRAWIOC3EPMrJBhBlxfAMxIETb0/H4rZEhPW22njDDLnca7if1bIm9kLnB", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDpduaNKhllWNs6faZ50AY0cTA+YiCiArlQlGUbSv3W4knswJ3ZEgld8jAKCTOOMmsai22UeiJkKj57Xh5ZVdj0=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIJABoIeQJwUd7aDRWaC+IV2a6JyxfGl/fN40EIr 2019-10-17T20:40:21.339 INFO:teuthology.task.ansible.out:gxTGT", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCvZvqdwAiqPH7Tv1SjnaVZbTB7C3O9xmcp8DXENfY2xFXll6gLqfr1/nmtdlJqRuSM68CE+wiFlpv/97UuOtY5a99TYfOzNqZL8qJxpfNs9wH12XDF2Jjyd9rLVwrH9pcARm4icefXOTSlntJt0YdOn/IJbjeeUulmAxLtMYzM/t4BD0TPrAi061MNPZamx99gWEZwZkN3FFTWHGNR+hKaZUqwSoFKGVxuSncL/Y/FJ27itleZRQTHTdPDbjeLe6e6yvgf/pLyAu/AvnIqrSyHBY+khT/2mYNqwfXY5mwZIBQRRGOI1RgRDxWuX7mp3oI5aCSE6BMFjw/NA5Dbv1RN", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 33, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-17T20:40:21.394 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"ansible_all_ipv4_addresses": ["172.21.15.204"], "ansible_all_ipv6_addresses": ["fe80::ec4:7aff:fe88:77ab"], "ansible_apparmor": {"status": "enabled"}, "ansible_architecture": "x86_64", "ansible_bios_date": "08/02/2016", "ansible_bios_version": "2.0a", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": "ttyS1,115200", "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_date_time": {"date": "2019-10-17", "day": "17", "epoch": "1571344821", "hour": "20", "iso8601": "2019-10-17T20:40:21Z", "iso8601_basic": "20191017T204021484297", "iso8601_basic_short": "20191017T204021", "iso8601_micro": "2019-10-17T20:40:21.484378Z", "minute": "40", "month": "10", "second": "21", "time": "20:40:21", "tz": "UTC", "tz_offset": "+0000", "weekday": "Thursday", "weekday_number": "4", "weeknumber": "41", "year": "2019"}, "ansible_default_ipv4": {"address": "172.21.15.204", "alias": "enp3s0f1", "broadcast": "172.21.15.255", "gateway": "172.21.15.254", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "netmask": "255.255.240.0", "network": "172.21.0.0", "type": "ether"}, "ansible_default_ipv6": {"address": "fe80::ec4:7aff:fe88:77ab", "gateway": "fe80::327c:5e00:6487:71e0", "interface": "enp3s0f1", "macaddress": "0c:c4:7a:88:77:ab", "mtu": 1500, "prefix": "64", "scope": "link", "type": "ether"}, "ansible_device_links": {"ids": {"dm-0": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi"], "dm-1": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB"], "dm-2": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN"], "dm-3": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf"], "dm-4": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfq 2019-10-17T20:40:21.395 INFO:teuthology.task.ansible.out:jO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB"], "nvme0n1": ["lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "sda": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "sda1": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"]}, "labels": {}, "masters": {"nvme0n1": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"]}, "uuids": {"dm-0": ["56e60058-f0cb-47e1-a3d6-bf071031bc28"], "sda1": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}}, "ansible_devices": {"dm-0": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi"], "labels": [], "masters": [], "uuids": ["56e60058-f0cb-47e1-a3d6-bf071031bc28"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-1": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-2": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 2019-10-17T20:40:21.397 INFO:teuthology.task.ansible.out: GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-3": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "dm-4": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "loop0": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop1": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop2": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": " 2019-10-17T20:40:21.398 INFO:teuthology.task.ansible.out:0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop3": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop4": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop5": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop6": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "loop7": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}, "nvme0n1": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory con 2019-10-17T20:40:21.399 INFO:teuthology.task.ansible.out:troller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}, "sda": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092941a36"}}, "ansible_distribution": "Ubuntu", "ansible_distribution_file_parsed": true, "ansible_distribution_file_path": "/etc/os-release", "ansible_distribution_file_variety": "Debian", "ansible_distribution_major_version": "18", "ansible_distribution_release": "bionic", "ansible_distribution_version": "18.04", "ansible_dns": {"nameservers": ["127.0.0.53"], "options": {"edns0": true}, "search": ["front.sepia.ceph.com", "sepia.ceph.com"]}, 2019-10-17T20:40:21.401 INFO:teuthology.task.ansible.out:"ansible_domain": "front.sepia.ceph.com", "ansible_effective_group_id": 0, "ansible_effective_user_id": 0, "ansible_eno1": {"active": false, "device": "eno1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2a", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.0", "phc_index": 0, 2019-10-17T20:40:21.402 INFO:teuthology.task.ansible.out:"promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_eno2": {"active": false, "device": "eno2", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "on [fixed]", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx 2019-10-17T20:40:21.404 INFO:teuthology.task.ansible.out:_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "all"], "macaddress": "0c:c4:7a:d9:36:2b", "module": "igb", "mtu": 1500, "pciid": "0000:07:00.1", "phc_index": 1, "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f0": {"active": false, "device": "enp3s0f0", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", "tx_gso_robust": "off [fixed]", "tx 2019-10-17T20:40:21.405 INFO:teuthology.task.ansible.out:_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "macaddress": "0c:c4:7a:88:77:aa", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.0", "promisc": false, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_enp3s0f1": {"active": true, "device": "enp3s0f1", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off", "l2_fwd_offload": "off", "large_receive_offload": "off", "loopback": "off [fixed]", "netns_local": "off [fixed]", "ntuple_filters": "off", "receive_hashing": "on", "rx_all": "off", "rx_checksumming": "on", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "on", "rx_vlan_filter": "on", "rx_vlan_offload": "on", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "on [fixed]", "tx_checksum_ip_generic": "on", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "on [fixed]", "tx_gre_csum_segmentation": "on", "tx_gre_segmentation": "on", "tx_gso_partial": "on", 2019-10-17T20:40:21.406 INFO:teuthology.task.ansible.out:"tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "on", "tx_ipxip6_segmentation": "on", "tx_lockless": "off [fixed]", "tx_nocache_copy": "off", "tx_scatter_gather": "on", "tx_scatter_gather_fraglist": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "on", "tx_udp_tnl_segmentation": "on", "tx_vlan_offload": "on", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "off [fixed]"}, "hw_timestamp_filters": ["none", "ptp_v1_l4_sync", "ptp_v1_l4_delay_req", "ptp_v2_event"], "ipv4": {"address": "172.21.15.204", "broadcast": "172.21.15.255", "netmask": "255.255.240.0", "network": "172.21.0.0"}, "ipv6": [{"address": "fe80::ec4:7aff:fe88:77ab", "prefix": "64", "scope": "link"}], "macaddress": "0c:c4:7a:88:77:ab", "module": "ixgbe", "mtu": 1500, "pciid": "0000:03:00.1", "phc_index": 2, "promisc": false, "speed": 10000, "timestamping": ["tx_hardware", "tx_software", "rx_hardware", "rx_software", "software", "raw_hardware"], "type": "ether"}, "ansible_env": {"HOME": "/root", "LANG": "en_US", "LOGNAME": "root", "MAIL": "/var/mail/root", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin", "PWD": "/home/ubuntu", "SHELL": "/bin/bash", "SUDO_COMMAND": "/bin/sh -c echo BECOME-SUCCESS-isceuiauaoqhkavjiswwuurwqzzmtixp ; /usr/bin/python", "SUDO_GID": "1209", "SUDO_UID": "1000", "SUDO_USER": "ubuntu", "TERM": "unknown", "USER": "root", "USERNAME": "root"}, "ansible_fibre_channel_wwn": [], "ansible_fips": false, "ansible_form_factor": "Other", "ansible_fqdn": "smithi204.front.sepia.ceph.com", "ansible_hostname": "smithi204", "ansible_hostnqn": "", "ansible_interfaces": ["enp3s0f1", "eno1", "eno2", "lo", "enp3s0f0"], "ansible_is_chroot": false, "ansible_iscsi_iqn": "iqn.1993-08.org.debian:01:31f958f4631", "ansible_kernel": "4.15.0-65-ge 2019-10-17T20:40:21.408 INFO:teuthology.task.ansible.out:neric", "ansible_lo": {"active": true, "device": "lo", "features": {"esp_hw_offload": "off [fixed]", "esp_tx_csum_hw_offload": "off [fixed]", "fcoe_mtu": "off [fixed]", "generic_receive_offload": "on", "generic_segmentation_offload": "on", "highdma": "on [fixed]", "hw_tc_offload": "off [fixed]", "l2_fwd_offload": "off [fixed]", "large_receive_offload": "off [fixed]", "loopback": "on [fixed]", "netns_local": "on [fixed]", "ntuple_filters": "off [fixed]", "receive_hashing": "off [fixed]", "rx_all": "off [fixed]", "rx_checksumming": "on [fixed]", "rx_fcs": "off [fixed]", "rx_udp_tunnel_port_offload": "off [fixed]", "rx_vlan_filter": "off [fixed]", "rx_vlan_offload": "off [fixed]", "rx_vlan_stag_filter": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "scatter_gather": "on", "tcp_segmentation_offload": "on", "tx_checksum_fcoe_crc": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "tx_checksum_ipv4": "off [fixed]", "tx_checksum_ipv6": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_checksumming": "on", "tx_esp_segmentation": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_gre_segmentation": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipxip4_segmentation": "off [fixed]", "tx_ipxip6_segmentation": "off [fixed]", "tx_lockless": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_scatter_gather": "on [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "tx_sctp_segmentation": "on", "tx_tcp6_segmentation": "on", "tx_tcp_ecn_segmentation": "on", "tx_tcp_mangleid_segmentation": "on", "tx_tcp_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_udp_tnl_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "udp_fragmentation_offload": "off", "vlan_challenged": "on [fixed]"}, "hw_timestamp_filters": [], "ipv4": {"address": "127.0.0.1", "broadcast": "host", "netmask": "255.0.0.0", "network": "127.0.0.0" 2019-10-17T20:40:21.409 INFO:teuthology.task.ansible.out:}, "ipv6": [{"address": "::1", "prefix": "128", "scope": "host"}], "mtu": 65536, "promisc": false, "timestamping": ["tx_software", "rx_software", "software"], "type": "loopback"}, "ansible_local": {}, "ansible_lsb": {"codename": "bionic", "description": "Ubuntu 18.04.3 LTS", "id": "Ubuntu", "major_release": "18", "release": "18.04"}, "ansible_lvm": {"lvs": {"lv_1": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_2": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_3": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_4": {"size_g": "89.40", "vg": "vg_nvme"}, "lv_5": {"size_g": "14.90", "vg": "vg_nvme"}}, "pvs": {"/dev/nvme0n1": {"free_g": "0.10", "size_g": "372.61", "vg": "vg_nvme"}}, "vgs": {"vg_nvme": {"free_g": "0.10", "num_lvs": "5", "num_pvs": "1", "size_g": "372.61"}}}, "ansible_machine": "x86_64", "ansible_machine_id": "e78bb49708c74f36a4e44b7cd5a3f26e", "ansible_memfree_mb": 31468, "ansible_memory_mb": {"nocache": {"free": 31782, "used": 272}, "real": {"free": 31468, "total": 32054, "used": 586}, "swap": {"cached": 0, "free": 2047, "total": 2047, "used": 0}}, "ansible_memtotal_mb": 32054, "ansible_mounts": [{"block_available": 226440784, "block_size": 4096, "block_total": 240095108, "block_used": 13654324, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501451264, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}], "ansible_nodename": "smithi204.front.sepia.ceph.com", "ansible_os_family": "Debian", "ansible_pkg_mgr": "apt", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-4.15.0-65-generic", "console": ["tty0", "ttyS1,115200"], "pci": "realloc=off", "ro": true, "root": "UUID=00033e24-d094-4573-8712-cc5f9717bdc8"}, "ansible_processor": ["0", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "1", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "2", "GenuineIntel", "Int 2019-10-17T20:40:21.410 INFO:teuthology.task.ansible.out:el(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "3", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "4", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "5", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "6", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", "7", "GenuineIntel", "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz"], "ansible_processor_cores": 4, "ansible_processor_count": 1, "ansible_processor_threads_per_core": 2, "ansible_processor_vcpus": 8, "ansible_product_name": "SYS-5018R-WR", "ansible_product_serial": "S188093X6928378", "ansible_product_uuid": "00000000-0000-0000-0000-0CC47AD9362A", "ansible_product_version": "0123456789", "ansible_python": {"executable": "/usr/bin/python", "has_sslcontext": true, "type": "CPython", "version": {"major": 2, "micro": 15, "minor": 7, "releaselevel": "final", "serial": 0}, "version_info": [2, 7, 15, "final", 0]}, "ansible_python_version": "2.7.15+", "ansible_real_group_id": 0, "ansible_real_user_id": 0, "ansible_selinux": {"status": "Missing selinux Python library"}, "ansible_selinux_python_present": false, "ansible_service_mgr": "systemd", "ansible_ssh_host_key_dsa_public": "AAAAB3NzaC1kc3MAAACBANMEck9GESxvxmlPQWWAc9+xWWX9ojaYleSG83jVU7pbOPa5kU4W3YyH//P/AG992WM8UqJZYn2ZLg0C58mCq0fSRfC5wJ6KHs3l+F/tEhBAekwPJnVYvAn5IKR/+cnepj+PWg2tlh2AU3Svnnlhvs8bNcinr+AA7YguVJRvK5oZAAAAFQDs4EBMn+x6HWk4zzRCO5PeGHusEwAAAIEAn1HXkqBFNVykrt51uPPrjrp2vWX3sD5PnWMvvG9Qovl0ufs9BypZEtQWw2yqac4EquTMltJUJpA+Vz5ilOeS/UlYsaQ2OQo6GokoOS5WBKphA3dnfSOWO4Wpewpp7T2eLZ553KXhtFz3JI25/ujPMu+beKexotnpXtcMQ/qqqNQAAACAAd2EY3J5qqF35pLkfFljo0akGaZZiJpm9Jt7fCpOa5IGKPFQuM0+vvYmE2HlAl6oC+/LsKdZHqiS8h98caZWgSKL3VvT211qCCLlPUJvXevF/MRac7A3Ja6mCU+Mn4/CPxJZgaCRXsbxFpmC7OFZFqzeQYrDwJGQ5I22KAX83PQ=", "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqsZalpKBKZAqxYwnXIF2yLGlgp/Xb3zi+PPJP9/u1OM7A9JToGvk7AJqt6m4SnGOK1WKXXgV7QUdikuQqZmH0=", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1 2019-10-17T20:40:21.412 INFO:teuthology.task.ansible.out:lZDI1NTE5AAAAIF8H48K0MbVIg4Oyt7fLXDjrNI4nR94y//Z98RLeF6Om", "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQCxcgkWcgpyXJpHPdcMl8MDrjKlUgM3Us9wefFAGIALtC4e2S4bWyY/L9DXlacl7CJCIIhy6oB7yzN/KZ35HJskpcwp6GAWk0PMRHnhOTC5xioN5fskgms3DKh5UXWqb9Ulyn4UWNRkKv20J5Lr43Jfo0qZ9QlSLRVf 2019-10-17T20:40:21.413 INFO:teuthology.task.ansible.out:+IrVLturRG49z2bZYvi2yapEuI28/HezE9qpx5Hd1L5qWQ4Fmjpz3XVjrjMgDduhjCEbyuQj5xGp8xFREZB1fYTfJ6JocCgqCfxH58cWTEB+c2ZFxV5HFrN9/VWLTtWlVaS6N4qNNmy1uQh/qUA94Jqb02P8Svi2/7YwIsYBhixu6JCBvGu9", "ansible_swapfree_mb": 2047, "ansible_swaptotal_mb": 2047, "ansible_system": "Linux", "ansible_system_capabilities": ["cap_chown", "cap_dac_override", "cap_dac_read_search", "cap_fowner", "cap_fsetid", "cap_kill", "cap_setgid", "cap_setuid", "cap_setpcap", "cap_linux_immutable", "cap_net_bind_service", "cap_net_broadcast", "cap_net_admin", "cap_net_raw", "cap_ipc_lock", "cap_ipc_owner", "cap_sys_module", "cap_sys_rawio", "cap_sys_chroot", "cap_sys_ptrace", "cap_sys_pacct", "cap_sys_admin", "cap_sys_boot", "cap_sys_nice", "cap_sys_resource", "cap_sys_time", "cap_sys_tty_config", "cap_mknod", "cap_lease", "cap_audit_write", "cap_audit_control", "cap_setfcap", "cap_mac_override", "cap_mac_admin", "cap_syslog", "cap_wake_alarm", "cap_block_suspend", "cap_audit_read+ep"], "ansible_system_capabilities_enforced": "True", "ansible_system_vendor": "Supermicro", "ansible_uptime_seconds": 38, "ansible_user_dir": "/root", "ansible_user_gecos": "root", "ansible_user_gid": 0, "ansible_user_id": "root", "ansible_user_shell": "/bin/bash", "ansible_user_uid": 0, "ansible_userspace_architecture": "x86_64", "ansible_userspace_bits": "64", "ansible_virtualization_role": "host", "ansible_virtualization_type": "kvm", "gather_subset": ["all"], "module_setup": true}, "changed": false} 2019-10-17T20:40:21.457 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.804) 0:00:07.895 ****** 2019-10-17T20:40:21.494 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-17T20:40:21.495 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.532 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.074) 0:00:07.969 ****** 2019-10-17T20:40:21.567 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.035) 0:00:08.005 ****** 2019-10-17T20:40:21.573 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-17T20:40:21.574 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.613 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.045) 0:00:08.051 ****** 2019-10-17T20:40:21.617 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-10-17T20:40:21.619 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.658 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.045) 0:00:08.096 ****** 2019-10-17T20:40:21.663 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-17T20:40:21.664 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.705 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.046) 0:00:08.143 ****** 2019-10-17T20:40:21.712 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-10-17T20:40:21.713 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.755 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.050) 0:00:08.193 ****** 2019-10-17T20:40:21.760 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-10-17T20:40:21.761 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.803 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.047) 0:00:08.240 ****** 2019-10-17T20:40:21.810 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-10-17T20:40:21.811 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.854 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.050) 0:00:08.291 ****** 2019-10-17T20:40:21.889 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-10-17T20:40:21.890 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:21.935 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.081) 0:00:08.373 ****** 2019-10-17T20:40:21.940 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-10-17T20:40:21.941 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:40:21.990 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:21 +0000 (0:00:00.054) 0:00:08.427 ****** 2019-10-17T20:40:22.057 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:40:22.098 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:22 +0000 (0:00:00.108) 0:00:08.536 ****** 2019-10-17T20:40:22.169 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-17T20:40:22.170 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-17T20:40:22.285 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-10-17T20:40:22.307 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-17T20:40:22.423 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-10-17T20:40:22.424 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-17T20:40:22.550 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-17T20:40:22.561 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-10-17T20:40:22.697 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-17T20:40:22.709 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-10-17T20:40:22.826 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-17T20:40:22.848 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-10-17T20:40:22.967 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-17T20:40:23.020 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-10-17T20:40:23.094 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-17T20:40:23.168 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-10-17T20:40:23.222 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-17T20:40:23.285 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-10-17T20:40:23.349 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-17T20:40:23.434 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-10-17T20:40:23.478 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-17T20:40:23.576 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-10-17T20:40:23.608 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-17T20:40:23.713 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-10-17T20:40:23.735 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-17T20:40:23.857 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-17T20:40:23.869 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-10-17T20:40:23.984 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-17T20:40:23.996 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/sbillah", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "move_home": false, "name": "sbillah", "shell": "/bin/bash", "state": "present", "uid": 1113} 2019-10-17T20:40:24.111 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-17T20:40:24.137 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-10-17T20:40:24.242 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-17T20:40:24.274 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-10-17T20:40:24.368 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-17T20:40:24.426 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-10-17T20:40:24.491 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-17T20:40:24.582 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-10-17T20:40:24.625 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-17T20:40:24.725 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-10-17T20:40:24.747 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-17T20:40:24.862 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-10-17T20:40:24.874 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-17T20:40:25.010 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-10-17T20:40:25.023 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-17T20:40:25.149 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-17T20:40:25.151 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:25.154 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-10-17T20:40:25.280 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-17T20:40:25.304 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-10-17T20:40:25.409 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-17T20:40:25.547 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-17T20:40:25.570 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-10-17T20:40:25.676 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-17T20:40:25.709 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-10-17T20:40:25.804 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-17T20:40:25.858 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-10-17T20:40:25.936 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-17T20:40:26.013 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-10-17T20:40:26.077 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-17T20:40:26.162 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-10-17T20:40:26.196 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-17T20:40:26.318 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-10-17T20:40:26.341 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-17T20:40:26.449 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-10-17T20:40:26.474 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-17T20:40:26.614 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-17T20:40:26.626 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-10-17T20:40:26.768 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-17T20:40:26.779 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:26.791 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-10-17T20:40:26.793 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:26.917 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-17T20:40:26.949 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-10-17T20:40:26.950 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:27.045 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-17T20:40:27.046 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:27.100 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-10-17T20:40:27.178 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-17T20:40:27.179 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:27.245 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-10-17T20:40:27.246 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:27.320 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-17T20:40:27.322 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:27.376 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jj", "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "move_home": false, "name": "jj", "shell": "/bin/bash", "state": "present", "uid": 1135} 2019-10-17T20:40:27.450 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-17T20:40:27.524 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-10-17T20:40:27.588 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-17T20:40:27.674 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-10-17T20:40:27.727 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-17T20:40:27.823 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-10-17T20:40:27.866 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-17T20:40:27.962 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-10-17T20:40:27.995 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-17T20:40:28.111 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-10-17T20:40:28.122 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-17T20:40:28.264 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-17T20:40:28.276 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-10-17T20:40:28.391 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-17T20:40:28.432 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-17T20:40:28.516 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-17T20:40:28.562 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-10-17T20:40:28.656 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-17T20:40:28.700 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-10-17T20:40:28.784 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-17T20:40:28.876 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhishekvrshny", "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "move_home": false, "name": "abhishekvrshny", "shell": "/bin/bash", "state": "present", "uid": 1145} 2019-10-17T20:40:28.919 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-17T20:40:29.024 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-10-17T20:40:29.048 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-17T20:40:29.153 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-10-17T20:40:29.197 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-17T20:40:29.282 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-10-17T20:40:29.336 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-17T20:40:29.431 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/shehbazj", "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "move_home": false, "name": "shehbazj", "shell": "/bin/bash", "state": "present", "uid": 1149} 2019-10-17T20:40:29.465 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-17T20:40:29.592 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-10-17T20:40:29.594 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-17T20:40:29.720 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-17T20:40:29.732 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/fche", "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "move_home": false, "name": "fche", "shell": "/bin/bash", "state": "present", "uid": 1151} 2019-10-17T20:40:29.858 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-17T20:40:29.880 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/onyb", "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "move_home": false, "name": "onyb", "shell": "/bin/bash", "state": "present", "uid": 1152} 2019-10-17T20:40:29.979 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-17T20:40:30.028 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-10-17T20:40:30.107 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-17T20:40:30.182 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/oprypin", "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "move_home": false, "name": "oprypin", "shell": "/bin/bash", "state": "present", "uid": 1154} 2019-10-17T20:40:30.235 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-17T20:40:30.321 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-10-17T20:40:30.375 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-17T20:40:30.473 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-10-17T20:40:30.510 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-17T20:40:30.617 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-10-17T20:40:30.642 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-17T20:40:30.751 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-10-17T20:40:30.786 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-17T20:40:30.876 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-10-17T20:40:30.933 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-17T20:40:31.032 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/asheplyakov", "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "move_home": false, "name": "asheplyakov", "shell": "/bin/bash", "state": "present", "uid": 1160} 2019-10-17T20:40:31.090 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-17T20:40:31.182 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-10-17T20:40:31.215 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-17T20:40:31.328 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-10-17T20:40:31.351 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-17T20:40:31.476 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-10-17T20:40:31.499 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-17T20:40:31.626 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-10-17T20:40:31.649 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-17T20:40:31.765 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-10-17T20:40:31.777 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-17T20:40:31.912 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/jwilliamson", "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "move_home": false, "name": "jwilliamson", "shell": "/bin/bash", "state": "present", "uid": 1166} 2019-10-17T20:40:31.913 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-17T20:40:32.038 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-10-17T20:40:32.060 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-17T20:40:32.196 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-10-17T20:40:32.220 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-17T20:40:32.328 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-10-17T20:40:32.350 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-17T20:40:32.487 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-10-17T20:40:32.507 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-17T20:40:32.636 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-10-17T20:40:32.649 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-17T20:40:32.795 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-10-17T20:40:32.828 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-17T20:40:32.949 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/liupan", "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "move_home": false, "name": "liupan", "shell": "/bin/bash", "state": "present", "uid": 1173} 2019-10-17T20:40:32.961 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-17T20:40:33.086 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-10-17T20:40:33.100 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-17T20:40:33.228 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/amarangone", "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "move_home": false, "name": "amarangone", "shell": "/bin/bash", "state": "present", "uid": 1175} 2019-10-17T20:40:33.231 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-17T20:40:33.355 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-17T20:40:33.367 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-10-17T20:40:33.476 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-17T20:40:33.533 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/kmroz", "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "move_home": false, "name": "kmroz", "shell": "/bin/bash", "state": "present", "uid": 1176} 2019-10-17T20:40:33.607 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-17T20:40:33.654 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-10-17T20:40:33.742 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-17T20:40:33.812 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-10-17T20:40:33.867 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-17T20:40:33.964 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-10-17T20:40:33.997 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-17T20:40:34.091 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-10-17T20:40:34.124 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-17T20:40:34.231 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-10-17T20:40:34.256 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-17T20:40:34.367 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-10-17T20:40:34.393 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-17T20:40:34.511 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/adamyanova", "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "move_home": false, "name": "adamyanova", "shell": "/bin/bash", "state": "present", "uid": 1183} 2019-10-17T20:40:34.522 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-17T20:40:34.669 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-10-17T20:40:34.680 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-17T20:40:34.806 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-10-17T20:40:34.818 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-17T20:40:34.945 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-17T20:40:34.978 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-10-17T20:40:35.073 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-17T20:40:35.129 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-10-17T20:40:35.203 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-17T20:40:35.277 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-10-17T20:40:35.329 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-17T20:40:35.424 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-10-17T20:40:35.477 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-17T20:40:35.571 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-10-17T20:40:35.624 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-17T20:40:35.718 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-10-17T20:40:35.761 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-17T20:40:35.850 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-10-17T20:40:35.892 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-17T20:40:35.986 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-10-17T20:40:36.019 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-17T20:40:36.134 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-10-17T20:40:36.145 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-17T20:40:36.265 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-10-17T20:40:36.276 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-17T20:40:36.401 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-17T20:40:36.413 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-10-17T20:40:36.531 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-17T20:40:36.543 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-10-17T20:40:36.661 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-17T20:40:36.686 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-10-17T20:40:36.814 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-10-17T20:40:36.971 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-10-17T20:40:37.108 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-10-17T20:40:37.257 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-10-17T20:40:37.259 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-17T20:40:37.405 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-10-17T20:40:37.532 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-10-17T20:40:37.681 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-10-17T20:40:37.719 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-17T20:40:37.830 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-10-17T20:40:37.968 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-10-17T20:40:38.108 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-10-17T20:40:38.211 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-17T20:40:38.699 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2019-10-17T20:40:38.752 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-17T20:40:39.129 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-10-17T20:40:39.213 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "name": "rakesh", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-17T20:40:39.637 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "name": "shyukri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2019-10-17T20:40:39.732 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "name": "ryg-", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-17T20:40:39.895 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:39 +0000 (0:00:17.797) 0:00:26.333 ****** 2019-10-17T20:40:39.925 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-17T20:40:39.964 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:39 +0000 (0:00:00.069) 0:00:26.402 ****** 2019-10-17T20:40:40.012 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-17T20:40:40.019 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-17T20:40:40.020 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-17T20:40:40.022 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-17T20:40:40.023 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-17T20:40:40.025 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-17T20:40:40.026 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h 2019-10-17T20:40:40.027 INFO:teuthology.task.ansible.out:7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6 2019-10-17T20:40:40.029 INFO:teuthology.task.ansible.out:c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa 2019-10-17T20:40:40.030 INFO:teuthology.task.ansible.out:2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db2 2019-10-17T20:40:40.032 INFO:teuthology.task.ansible.out:93dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.red 2019-10-17T20:40:40.033 INFO:teuthology.task.ansible.out:hat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f 2019-10-17T20:40:40.034 INFO:teuthology.task.ansible.out:57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-17T20:40:40.076 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-10-17T20:40:40.078 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:40 +0000 (0:00:00.111) 0:00:26.514 ****** 2019-10-17T20:40:40.145 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-17T20:40:40.145 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2019-10-17T20:40:40.607 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "name": "rakesh", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2019-10-17T20:40:41.008 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:40:41.013 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-10-17T20:40:41.014 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-17T20:40:41.056 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:41 +0000 (0:00:00.979) 0:00:27.493 ****** 2019-10-17T20:40:41.158 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-10-17T20:40:41.159 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "name": "ryg-", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2019-10-17T20:40:41.265 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:41.266 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-17T20:40:41.316 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:41 +0000 (0:00:00.260) 0:00:27.754 ****** 2019-10-17T20:40:41.339 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-10-17T20:40:41.379 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:41 +0000 (0:00:00.062) 0:00:27.817 ****** 2019-10-17T20:40:41.415 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:41.416 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-17T20:40:41.429 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-10-17T20:40:41.434 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-10-17T20:40:41.436 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04 2019-10-17T20:40:41.437 INFO:teuthology.task.ansible.out:okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "jos 2019-10-17T20:40:41.439 INFO:teuthology.task.ansible.out:hd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jo 2019-10-17T20:40:41.440 INFO:teuthology.task.ansible.out:wilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f 2019-10-17T20:40:41.442 INFO:teuthology.task.ansible.out:966b2794c94caa110d313e927a1c1b\n"}, {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h 2019-10-17T20:40:41.443 INFO:teuthology.task.ansible.out:7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6 2019-10-17T20:40:41.445 INFO:teuthology.task.ansible.out:c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa 2019-10-17T20:40:41.446 INFO:teuthology.task.ansible.out:2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db2 2019-10-17T20:40:41.448 INFO:teuthology.task.ansible.out:93dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.red 2019-10-17T20:40:41.450 INFO:teuthology.task.ansible.out:hat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f 2019-10-17T20:40:41.451 INFO:teuthology.task.ansible.out:57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}]}, "changed": false} 2019-10-17T20:40:41.506 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:41 +0000 (0:00:00.127) 0:00:27.944 ****** 2019-10-17T20:40:41.544 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:41.546 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-17T20:40:41.695 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-17T20:40:41.821 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-17T20:40:41.823 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-17T20:40:41.957 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-17T20:40:42.092 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-17T20:40:42.094 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-17T20:40:42.115 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:40:42.125 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-10-17T20:40:42.126 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"after": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "attempts": 1, "before": "969743e3a31f8a69a040415d87f1ea3564ff39fb", "changed": false, "remote_url_changed": false} 2019-10-17T20:40:42.169 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:42 +0000 (0:00:00.662) 0:00:28.606 ****** 2019-10-17T20:40:42.238 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:42.238 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-17T20:40:42.353 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-17T20:40:42.365 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:42.397 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-10-17T20:40:42.502 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-17T20:40:42.503 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-17T20:40:42.513 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:42.514 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:42.525 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:42.536 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:42.614 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-10-17T20:40:42.657 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-17T20:40:42.761 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-10-17T20:40:42.804 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-17T20:40:42.805 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-17T20:40:42.896 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-10-17T20:40:42.928 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-17T20:40:43.033 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-10-17T20:40:43.034 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-10-17T20:40:43.055 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-17T20:40:43.169 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-10-17T20:40:43.191 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-17T20:40:43.317 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-17T20:40:43.339 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-10-17T20:40:43.340 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-10-17T20:40:43.479 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-10-17T20:40:43.480 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-10-17T20:40:43.616 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-17T20:40:43.639 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-10-17T20:40:43.656 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:43.781 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-17T20:40:43.783 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-17T20:40:43.793 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-10-17T20:40:43.794 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-10-17T20:40:43.809 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:43.810 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:43.821 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOt 2019-10-17T20:40:43.823 INFO:teuthology.task.ansible.out:sItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:43.915 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-17T20:40:43.916 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-17T20:40:43.985 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-10-17T20:40:44.049 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-17T20:40:44.122 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-10-17T20:40:44.123 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-10-17T20:40:44.185 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-17T20:40:44.277 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-10-17T20:40:44.309 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-17T20:40:44.311 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-17T20:40:44.422 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-10-17T20:40:44.465 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-17T20:40:44.572 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-10-17T20:40:44.594 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-17T20:40:44.707 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-10-17T20:40:44.728 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-17T20:40:44.865 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b1 2019-10-17T20:40:44.867 INFO:teuthology.task.ansible.out:55b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-17T20:40:44.990 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-17T20:40:44.991 INFO:teuthology.task.ansible.out: ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-10-17T20:40:45.116 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-17T20:40:45.118 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-10-17T20:40:45.149 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-10-17T20:40:45.151 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-10-17T20:40:45.245 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3kZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+ 2019-10-17T20:40:45.246 INFO:teuthology.task.ansible.out:oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-17T20:40:45.279 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-10-17T20:40:45.281 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-10-17T20:40:45.397 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-17T20:40:45.427 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-10-17T20:40:45.522 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-17T20:40:45.565 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-10-17T20:40:45.650 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-17T20:40:45.708 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-10-17T20:40:45.715 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-10-17T20:40:45.776 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-17T20:40:45.852 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-10-17T20:40:45.906 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-17T20:40:46.000 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-10-17T20:40:46.037 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-17T20:40:46.144 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-10-17T20:40:46.155 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-17T20:40:46.257 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-10-17T20:40:46.289 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-17T20:40:46.404 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-10-17T20:40:46.408 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-17T20:40:46.548 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-10-17T20:40:46.550 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-17T20:40:46.685 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-17T20:40:46.686 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXvd7F0n4DeBROTLYDE4lItYn/2S1cMM1uAkkwissUh4LGsMAVWi5tUI0EUC7I2BUxACvIIA4yfkhe2ATAu95UQxKWVql67sfCW6WwY7lNa65kQC9S5dE/+d4vPxFoFSFnri981laJXOjbjTsWD7j1SGfqbRuAHQKL8y95DQ3y/B7/1oxTg1jjK8BTr3GAUp8/7lNlWE+x2S3gjHqqH92yWU7OBbPWRZjjkVkXS/5CbWDDUx8nbeGPcAIfM95rRQR0VkwzVi1FWsXd6qIbZ9TqOhYyLR0fCsnDuveNV68FB1ddvqCLdXRxPLmucwzdjGMbZ5EJ0H/CAAutj3Vjn6gh jj@newdream.net\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRXOkus/WHB3llEH1u4rWj0F/bZSslq4/o7pBUGFYrHKqa2iBiCj4DxBEsaReWUm0aDyesGuEVeoECTtzh7byJ6iZHFbf0McubxnuFMImneQ2mB0+raa3uWblwnyiwAqiZgtcJGQpCxCnrmLzyNN6xUUmzN27cDOMSDpbWr3VdPSBH+ixIuZI8r8UkSbmlX30V2bvk4lGl2SPV385tAvRPRU9KfMG9AgwNv+R2WCiA2GOrjes2VHm7LneqI/vE5yOxJsm4tPxEkOpCp3/0xt9L9cmPz47HnzJ2Qure3k 2019-10-17T20:40:46.688 INFO:teuthology.task.ansible.out:ZmOsCxOX1+9dXFIwhPaNconD4KDlv4MNs/M425Gp6KKnnV jj@laptop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBjrQ3drodiPr83uvpUUf6SXZ0utOFIfy5Cj3BGcn7bY29BKXCwomAUmmLMk76cGBZ7jHf3ZJYid1ESbHmvmSgKFShr9JFXBHvlMdnyewi7hFix+sjX9ZtuZfSaIsBbU5Jlz3RqHtDrScS1wJRh+oDlH6is5QWSCABVWhUk+hJtQFg4dfespqomt88NmPZuR0aR0b22hkrMYINXP0y3iz3Jjs5vg8td7VJ2hXb5yIBk+Izuq1LnjPyRCZv5LkZMuwofexpfflLKeVHw0t31HXuzBUZMoKqXiGJweyPDzY/xbeTVnYvQQEvoGxRDpaYuiGSfhFW0UHE3HSeuYfklO2x juanjose@aurora", "key_options": null, "keyfile": "/home/jj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jj", "validate_certs": true} 2019-10-17T20:40:46.820 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-10-17T20:40:46.821 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-17T20:40:46.822 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-17T20:40:46.958 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-17T20:40:46.960 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-10-17T20:40:47.117 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-17T20:40:47.129 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-10-17T20:40:47.274 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-17T20:40:47.275 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-10-17T20:40:47.412 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-17T20:40:47.434 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-10-17T20:40:47.549 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-17T20:40:47.612 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-10-17T20:40:47.677 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-17T20:40:47.761 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-17T20:40:47.804 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-17T20:40:47.908 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-10-17T20:40:47.930 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-17T20:40:48.056 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-10-17T20:40:48.067 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-17T20:40:48.069 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-17T20:40:48.070 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-17T20:40:48.209 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAfEhx/xZpJV9ouKFW4QBC46cNicIQMzU8DSYl1GOP9+pnnD+AudVQubvrEePGA7TFkSar5Ccv4dnoQ2s1CaPAtbT+s+nQlf5OFO9P/qvhkM4JhlmH2XxgJAcOvZH2W1HXfGTc4WIuU/OtXM+yWMGlTXxqMqwCvGfqc+Yf8zkt0t3m0kBdWoFcZvdYqKuS15bvdVuiKz7GUxDrtw98yMXb/ob/E+Ygmmphn9dMatTeZ6SD1fbcrHh9oIVbYiY93+++jXzjEXrGzJh/gphgEtvIf8UPMF9HiXXPqwhxneRvghvZ/B05LGfcsqGAtyK09eF94hglyl9kJIdm/d7lmmM/ abhishek.varshney@VS-C02PJ2CSFVH5.local", "key_options": null, "keyfile": "/home/abhishekvrshny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhishekvrshny", "validate_certs": true} 2019-10-17T20:40:48.220 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-17T20:40:48.356 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-10-17T20:40:48.358 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-10-17T20:40:48.368 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-17T20:40:48.514 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-10-17T20:40:48.516 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-10-17T20:40:48.525 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-17T20:40:48.674 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-17T20:40:48.676 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-10-17T20:40:48.820 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUqFGbkDz4L4BpIs2znPSWeawZZM2FBLrR3htaMBM6lcmiX2h+vgVGv2ZLpajGo7KB6J59kzWLBCJXN2DG0uUqa7dOdjEDuDrjZkNgDzXZ20lqzFAjLwNF6a9PcDN/oX6jzK9+3xArleQuTIJsOPRue2dQYGRnyIIrYpapJkkUgI+lXfwrEixCY3xjGNvVoh5bqQmkk3buLIvKEMwhPXJ94SGqsNTNC75HfKhVMjxA8QmuiXJVEgiW00V3Cctwv+yjiDMts4Dcs6yEzxXzjv0c0APxxN78bKlkwFgXiqawLiApk752/x2teSg7kvdrlGcCAWq/BhwphYjwaRlXIj1N shehbaz@donjaffer", "key_options": null, "keyfile": "/home/shehbazj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shehbazj", "validate_certs": true} 2019-10-17T20:40:48.821 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-17T20:40:48.958 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-17T20:40:48.969 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-10-17T20:40:49.085 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-17T20:40:49.117 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJorrIT68EZc7A4IHwQGv8uoblSAIUnMxhOq6rlnvmqx4clpUzKkbzzrZUctP47aeqaSX5Mc6mV8TY0sIH8fFQwahF3ueTKT3YecZvPwiK4ALmawkvFpmgnJKKYhTrEApbFBhtH0KnNLtozpl4RJds4p09maNeoUXJDy3A24tpXViU2Nrw8lSIT6VJzvqSlK+miNEf3VKI6gnRB7fGcJe913zITf09Rew7GexAyQUYHusZbJCf5rl6hkewkZMkgbXfG22f/XWcYjoRtA/ZqeWIP2UDO/H3tfJnlMxUshbDGLllMFJzwmwhkYbapqVmnu7Y01rEQ4LqvaHZMr/yKn+N fche@very.elastic.org", "key_options": null, "keyfile": "/home/fche/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "fche", "validate_certs": true} 2019-10-17T20:40:49.222 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-17T20:40:49.285 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmKchKXS6xKpo+25ZN5vMpbuL4OoFZWPfPSxur6iaBKdr6LSMejnAwkJOhfhH55pZLuAglHWdMcZi8H/ktzsZVd+ybvJNhpZsXdefNqqxUyjn320cT73pDdDC/W1519fUiokss23S6B4GR0B44VrVKezN5Dyhf4sATN8l76QnIALVK5hvYOI3v5k70weaPJM4fma7xYF5b6qSxauGofzMb5XwjNaaPE9F8U08nxGk5XK+S0rHoFnCjoKBlN9Ee1gk6PjsStQAyK5ujWvPGtG/UvuU9LewVbt6wuPbpwsf7Ck0u4wHECHK+185JF/X+qsWwv030pKShXblLVckNVhAx ani@orchid", "key_options": null, "keyfile": "/home/onyb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "onyb", "validate_certs": true} 2019-10-17T20:40:49.348 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-17T20:40:49.423 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-10-17T20:40:49.475 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-17T20:40:49.550 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzf8fjotuelTIdz7clDpSN/VDtXirYrmzM/r157Np9uTMgn/rhzzplp2DKqFrCEDwawJ1O1Fr/H5DqttPonyRmjPmAIf96P8KKLsC600R63xk4DNhnByY0WBU3o1e46LP7DJLsOOy6hZRbyjgoFfzcapH7aV+fa37k7/Ri5mpmdmENa6lYI38+YaItFJf0kTXdFou0iLAt7xogOTimgblMst6UjOhRIsMFA3QS4PvOFR+AedCHeVtYK9mAeRtQ530wKzjz8x4NFU3Y93jmygwTTG2RTEfRjfbObfFOQXJGKcl28pxGPUL9ZlrLx1juAekn2Uz+2oO/eQ8G5w+XW3rp", "key_options": null, "keyfile": "/home/oprypin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oprypin", "validate_certs": true} 2019-10-17T20:40:49.624 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-17T20:40:49.625 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-17T20:40:49.677 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-10-17T20:40:49.751 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-17T20:40:49.834 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-10-17T20:40:49.836 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-10-17T20:40:49.837 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-10-17T20:40:49.876 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-17T20:40:49.970 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-10-17T20:40:50.003 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-17T20:40:50.005 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-17T20:40:50.109 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-10-17T20:40:50.131 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-17T20:40:50.257 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-10-17T20:40:50.268 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-17T20:40:50.384 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDLCJHta/5O2C2MXRegwo+bJUiqbkESr+RwoH/VIdb4L0JDlVsqV+K+BOBjLZy3vwzWCaSackvshoZHAwGvvfGXjtLbhAYs6QWWHODf/XWBvOW4a96fbTToNpV1dXkHAGqlJMYln3GKVD0o9b69iQUTcLILJhWQjCoTjZsvTtP9OYn9YLfuoxhaaU3PazZVhZYml4Ca/3ns9vNQ/m+v4nh82ysyflGKYozEq0pl8VH0Ydy/FtOO98PhQx3crrivu3qghubntVRQnULUmlDSFhMff+ukLVdj2U0AdSBatlJuaSofWUMAQkv0Ia/om9doDGv2TqkXhJnZdGaWJwmuh6rKIZr7d7TYTD6Uag3wJ1NKiDETRNtmMCLuniEJzPO4asBpSSWGyVatyrNmAgBb6XCeZRkRcYvTXdA5OESKG3HJeguk0XCrjosIEYcvxyye9SgA8U73jWdDUdh0yIHLWcfrgC2eynE1uBnTPNBvtnf9pKqMEp5opdRAUhckhPRxJQ2JHcYMAFnq7gz1bL2boL2ElZOqHBx9qgv0ZKnk+YXqhyp2EP2JK0urd0EHMqkGUsRc7D0FSrz+j4lSkd71sHu3tMgwIrgDR55wYMBBRXTrR8pGTMnGldDBaxvUCgWnlsTQJdjs9xCM5gcJzzgZgkRBmrTOUkw2ABre6KMRAmQ4Zg3HxO0MBr5rfV5glWXuDJXsEowLPp/NQ+3X7PIU9LAtKkcZPWC3MayoFdEdNpKieCgOfRAiwJ34fWz4ozM0IvENmUmCKz8+yDAOeupJJ1/367Rl3+U8XTsDNSsqcmsRdr8ZnUIN0662FGr/nHkVaDq79rEywt/V2eMLWyunF0FUUnbwBB8/J0hgIQPX1ncziuzPTF/YCTR6lgVEtvNHPI1xTQsaVQkF9Whu9Eq6Gz1cFq6H/abb1DLJAoO1Cy0HjoT8MjwOICVcsrqxGXsNRBsv4fCxOaz6+WyVWz2O5gpNKOLLmenr9JbiTGjamjfl9dDempRROSDjsJJY6TPdRb0WBuk3i6mOAsAYeBHoargi+a+dQ1IDzGdVY2mUXtDTA6HluZVvzEKnAR6QzuEdRLOrbteETRQ3cV9STXgscCTokzDNzhYbkLUIl/SRBYBdPMiR9TgW+m/0IJkBTEEUuDtYVzyHezV94F17vzpK+xbRdk3/WmWPScPDkc9tDp0AGLdZ/FAkNzGM1VE8G5OJ7OduogIdIQW6yR8y+CP1O4e2WYZ3M0IjJFX8pc/zBinUBNOBvgAzO8aSXTgxXbs/Q1fcv7htmJZ/3nnB4U6N85tq5oklCWulk3tIFI5pMGiQUoQCW8PH2nwasaKKtOInBZMWm8B2K0c5bU2/bKRcXJNJ asheplyakov@as-ubuntu", "key_options": null, "keyfile": "/home/asheplyakov/.ssh/authorized_keys 2019-10-17T20:40:50.385 INFO:teuthology.task.ansible.out:", "manage_dir": true, "path": null, "state": "present", "user": "asheplyakov", "validate_certs": true} 2019-10-17T20:40:50.395 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-17T20:40:50.512 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-17T20:40:50.534 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-10-17T20:40:50.650 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-17T20:40:50.671 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-10-17T20:40:50.786 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-17T20:40:50.808 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-10-17T20:40:50.923 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-17T20:40:50.976 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-10-17T20:40:51.049 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-17T20:40:51.102 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-10-17T20:40:51.174 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-17T20:40:51.227 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCmp/WxBoQxK/3bI3N9X4INmChIFtG/75NULYt9CAAnoOhFHe8xHMOU4x/YS6JdoIBJXusrjbDhCksFSK/caoOow9CQp3VWytslRSjfjkjbwwWHSdcyGpSO4lS5F32SgLOZBuyktpkLDgi6Ou5OTNKokJO8qiZo9Uw0rEBNakfsVi+fpysUDL792Jk1ROUr0HUpwqrQfTPZaue+PomXPQSYIqf06EH1nJYho6D4jt/g6U0qc66WTW5dXTkWD2VVQjZbFGkamz0ICHBJ/Xpf4TjWQVotsYD5NtbvqPo6ygpJYNhKR2n2zA+L4Zu2n0AEib5S4ftNxc7gNDCCfVs/EEIvXp/3CjKVPxHf4xUUPUZj+SQFnSUPQfeY1iIydoX+IG9oj5kKmQBdQFxbAaZJ679bORIqlH73dUFGr2rfS1UlIgPW5ll77bixsB0GP3+Jw4FF5QbO+k1d5qJpUw+8D6FbWAppP7hnSAok9PTpBqfkMpFiX00ZTIFwKCBLxdtJOfbIhbOPCnyEoB2ZLfRiBJmXhBpLjGghWIPPnGZh+yeJiy/EyKwN57vLmjdmvcDgSsjHswT+rQ+QzPoF+/KcdU1LL6ME6i+R7o7aiLOdDpZo4p5d1kr3VMNjVNM+L9ECpTntPhyrglOX018Q8xcNRh2/z+G6NsiiV4t0YxFdk/ICTQ== Jesse.Williamson@suse.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCXsBAd8JfyMT352SjfhZ2Nj3gR2PkeyPS9yB6fKck5CW0Ns1EcUAOIFYw7a1R6zCDJbkkXZDRgM1W6v/UBgpOm69HaNCZuqx19mQRNgo1lhw6hDHc+9754wOmIbgu3gInIS/XPvGzAkGwBW3rWlOaNurY4Y/PH1QyiaujD1Ii9X2Q6yRP8hsDrccual36gngY6wV3G4xKbS8kTY4qQekHkgFT+ZDHv9ibliDfAxtzm5X8N25sICpvcnQkG+Co3QP2XADEhoQrs8L7QekKYPlX8zq6rJBO0cGYW50E58LJk7s67n7QQPCy04xeAf9KOZZBSMHTI8eSFHjb8B034IkHH jwilliamson@linux-z0c6", "key_options": null, "keyfile": "/home/jwilliamson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jwilliamson", "validate_certs": true} 2019-10-17T20:40:51.311 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-17T20:40:51.378 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-10-17T20:40:51.380 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-10-17T20:40:51.431 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-17T20:40:51.505 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-10-17T20:40:51.564 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-17T20:40:51.647 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-10-17T20:40:51.700 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-17T20:40:51.796 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-10-17T20:40:51.798 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-10-17T20:40:51.817 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-17T20:40:51.922 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-10-17T20:40:51.955 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-17T20:40:52.080 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-10-17T20:40:52.102 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-17T20:40:52.224 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-17T20:40:52.235 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC3aWYoBDxRh13e+qver2xcmCtNsiVEt9NkN1cntdUxVbjfWgb+E/b54kPW456tswoVaYS+3xI5DYTPTN1tMhx8jnrgVKI2HhIYC3SO0+V+HNofzU1p7v5CfZxwyupXOgdfHmEWKOepPop8qWon8wZVUkj2t80w8Ifxh5gLVigGquc4kvMLyy6nBI0k6UGeRr9lSKmWLvzudWP70cWij3ynwxEeIuurU0HI9tDmjaVW5dJuu6IZHDhJrCpeUZBQIDqjH0pg1i1uXXwwiLI9NW8jWuALREDeMxAG4IgOZtoXdS7J9vVXHxKpa8ofb+qYNXI7B3OTLgjSS2ziiU1KEIr wanjun.lp@alibaba-inc.com", "key_options": null, "keyfile": "/home/liupan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liupan", "validate_certs": true} 2019-10-17T20:40:52.360 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-17T20:40:52.372 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-10-17T20:40:52.487 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-17T20:40:52.540 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0PusmXbjqduPTosGmqFsi2cUjRIiWmMnATY/vuawopBxRNykGnyeh1MfVLgkulM6zBJ3cDZxLsZYykpsHGvaXgLL5r6bEOOKPu8sSMwaTfZWNewue6xa1vOYAggeWOt8KTd0IO/pZZfvFcJ80tjtcNvY8zOJ9TO5W7RpglfJhPneC9Kr726yZ4RX3Hqlx+nIMzrSlvZVtwsKikanImvGLDulRiHqgE5BcqGvDF1yCSIq4OYF2i2Sk+dnPnQtN2uePsxPYKikd0B/rua5q+hDDAz2I36NhaUIhLteE6acWJK5Z2s0o1rWAS5XMbF4qYM3nbaE1Au/nXHcjO3sf6mj3 amarangone@macair.local", "key_options": null, "keyfile": "/home/amarangone/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amarangone", "validate_certs": true} 2019-10-17T20:40:52.614 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-17T20:40:52.709 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-10-17T20:40:52.753 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-17T20:40:52.834 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIGGibBaZykvZPLLQm/UIDUlqVxeyJvZEOO+/7pjqX3O47nilNILLnv/oWhQN1S5eGVev9lnTlTxqI0Mt0N/B9vbCxoxx3ujsSgc+FZL58rMBN4uaTvHhYgqOYo3ht6fciRqSvcqeOGNKU8HnXS0/C8bJrYoKUTnmaaZjQ0bSw/AMsAIxu3bZmWT+HfgVaMh9PjPgC+XnDypEcRkOLbeEIs+0s1TgSR7Hs/vEuVtLH3Ph4WZwKVKQhfugmbblLLNvHvTy0q5afdh2RQ9MEaUgxXMgVq4hJ/JcFNZp+ZU5irOPQosZfjSLWJKx/LH+Ye4/8nINi08w0MZgcK79CwukJ kmroz@suse.com", "key_options": null, "keyfile": "/home/kmroz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kmroz", "validate_certs": true} 2019-10-17T20:40:52.898 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-17T20:40:52.981 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-10-17T20:40:53.024 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-17T20:40:53.107 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-10-17T20:40:53.150 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-17T20:40:53.244 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-10-17T20:40:53.276 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-17T20:40:53.382 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-10-17T20:40:53.414 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-17T20:40:53.530 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-10-17T20:40:53.541 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-17T20:40:53.668 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-17T20:40:53.669 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-10-17T20:40:53.794 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-17T20:40:53.812 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDf1IrqEe8zPM6DWf8Y+WLRU+TwX8Xzki1CQDRwK/w/F7qxAuh9cbX8tCTkZUt0L3pUjGpM1C3bkTzLHaN4bxK1RQ/klB7C3AXjNOJSPJSjcn/N3YuKU3sKWuQkXnQTwqUdTzHuZ8lQWywFItmwQR2Y6T+xG/Ecig+vdU5lwQ479gtt/lB9buI+T10Lefk2bUiYq1PmKSCIF4KtsIdmSazmIQyvsbsBVblMEezCC0auwo7P7SdfSaobf4x4ZZyJfYkVxiL6eGJ8vmBSbJFrcVMkAO07wBtpn/gQa3Dt4xUdXXt7zBizpMmRPLm8m3Pl2kIJhqrIa1cFXPRr5GJ0YXzR mu3e@Mu3e\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNisR+EK2sG/E9dSeoEYCfgzIunFnuu+SFWGyWR3K6Q5Evt7bVAsGnjglNXdVgLbGV4Il+Z+FFgtR/oJgtUZ7G0IwWKG+nlx06hOu2DObsgM7gcIlhrf7SNsI13Bit0v5dQkyLwcmpdwioQyUx5hnx8UamqVo7lLtKsHS8melz7JE/+dnGwIMJLB1Ot/8PIRAb+/NRXpnD/lje5u8idr43bhreuJ6Lell+h8X2vomuvLlA5XjAoROY6L5qKoM3+bD4QXbU006VHg/7wl9tpZIi9XwAuHfaktsmfks+u3eYFCd9nYUKN20zbEecoAA1rOrCF1qhgwTYpa/Rmh7Iy53p mu3e@thinkpad-t510", "key_options": null, "keyfile": "/home/adamyanova/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adamyanova", "validate_certs": true} 2019-10-17T20:40:53.918 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhos 2019-10-17T20:40:53.920 INFO:teuthology.task.ansible.out:t.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-17T20:40:53.964 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-10-17T20:40:54.066 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-17T20:40:54.068 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-17T20:40:54.088 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-10-17T20:40:54.192 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-17T20:40:54.204 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-10-17T20:40:54.319 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-17T20:40:54.341 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-10-17T20:40:54.458 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-17T20:40:54.482 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-10-17T20:40:54.608 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-17T20:40:54.629 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-10-17T20:40:54.746 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-10-17T20:40:54.757 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-10-17T20:40:54.880 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-10-17T20:40:54.882 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-10-17T20:40:55.018 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-10-17T20:40:55.020 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-10-17T20:40:55.156 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-10-17T20:40:55.158 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-10-17T20:40:55.272 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-10-17T20:40:55.284 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-17T20:40:55.421 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-10-17T20:40:55.422 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-17T20:40:55.547 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-10-17T20:40:55.558 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-17T20:40:55.684 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-10-17T20:40:55.708 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-17T20:40:55.832 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-10-17T20:40:55.854 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-17T20:40:55.958 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhos 2019-10-17T20:40:55.960 INFO:teuthology.task.ansible.out:t.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-10-17T20:40:55.990 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-17T20:40:56.092 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:56 +0000 (0:00:13.923) 0:00:42.530 ****** 2019-10-17T20:40:56.100 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-10-17T20:40:56.101 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-10-17T20:40:56.126 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.137 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.149 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.150 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.161 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.162 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.184 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.186 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.189 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.252 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:56.253 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEJa/U9zUfZ7YFl0kwU7Mx3rHPn2O4sdh1J3XBUgoVdrJ4Qnh7i+QhN+fw0eaBWQE2GIsGEX6YWAL5xJ7EuYZyas8OVLNdP9cJIr79M51d8dG/ywqBpgcFnx7qhCSua3sKmP6GCTWGD9Y0eWL2Z6y9t3amb8dPXviMNqKGDz/DaDtp5bPaoG8mDHQ9QeADYYk3tx6V7+phIBXrI2GKt0BBlJeuZysiUI/XWO46JdMyHvCzvzcdquD446aoWS44waDGj501sgnHVQ7gwf4P1+vFOSEHzkWhbenpJmmTA26f4dim+Jo5JpBW/Ma5mMBvORFhH/Kt+S9P9sPCiF1shWc3 d@d", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-10-17T20:40:56.315 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:56.317 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-17T20:40:56.327 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.412 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:56.413 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-10-17T20:40:56.445 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:56.446 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-17T20:40:56.565 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:56.567 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-10-17T20:40:56.587 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:56.589 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-17T20:40:56.708 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-10-17T20:40:56.715 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:56.716 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-17T20:40:56.851 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:56.852 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c7 2019-10-17T20:40:56.854 INFO:teuthology.task.ansible.out:0f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11 2019-10-17T20:40:56.855 INFO:teuthology.task.ansible.out:c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.865 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.865 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.876 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.877 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.888 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.889 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.903 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.911 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.923 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.934 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.937 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.946 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.957 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.968 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.968 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.979 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.980 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:56.982 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\ 2019-10-17T20:40:56.983 INFO:teuthology.task.ansible.out:njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.992 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:56.993 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.004 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.005 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.016 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.017 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.028 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.028 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.040 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.041 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.052 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.065 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.067 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.068 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.071 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:57.072 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.084 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.085 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.098 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.098 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.110 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.111 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.122 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:57.124 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.135 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.135 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.147 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.148 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.160 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.161 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.163 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.173 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.174 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.184 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.185 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.197 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.198 INFO:teuthology.task.ansible.out: 2019-10-17T20:40:57.200 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.209 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.223 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.233 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.235 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.250 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.251 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.252 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.264 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.266 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.268 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.287 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": 2019-10-17T20:40:57.290 INFO:teuthology.task.ansible.out:"item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.299 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.300 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.311 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.325 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.327 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.328 INFO:teuthology.task.ansible.out: skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.339 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.342 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.353 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.355 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.366 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.367 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.377 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.388 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.389 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.402 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.403 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.415 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.416 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.418 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-10-17T20:40:57.420 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.429 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.440 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.441 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.452 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.454 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.465 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.466 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.477 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.478 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.489 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.502 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.578 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:57 +0000 (0:00:01.485) 0:00:44.015 ****** 2019-10-17T20:40:57.584 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:57.587 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-10-17T20:40:57.609 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:57.652 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:57 +0000 (0:00:00.073) 0:00:44.089 ****** 2019-10-17T20:40:57.695 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-10-17T20:40:57.805 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:57.807 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-17T20:40:57.843 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-10-17T20:40:57.933 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:57.935 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-17T20:40:58.017 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:58.019 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-10-17T20:40:58.049 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:58.051 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-17T20:40:58.144 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-10-17T20:40:58.145 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-10-17T20:40:58.226 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:58.228 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-17T20:40:58.279 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:58 +0000 (0:00:00.627) 0:00:44.717 ****** 2019-10-17T20:40:58.304 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-17T20:40:58.327 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:58.329 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.330 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.341 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.352 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.353 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.365 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.376 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.377 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.419 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:58.421 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-17T20:40:58.505 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:58.505 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-10-17T20:40:58.516 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:58.554 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi092.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-17T20:40:58.664 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-17T20:40:58.665 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:58.667 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-10-17T20:40:58.781 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:58.782 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-17T20:40:58.792 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:58.793 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9', u'name': u'sbillah', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "name": "sbillah", "ovpn": "sbillah@syed-machine qVcw+LuFQQxYW7QpzZ3aLA d028c4635289a781f3ebe26a545e084572613b03cc9cde7770018ad0259e4dc9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEgeHHS5jx9k7QdOEGqZjaxEAPJ6vv/WZXJNpifBpm9Ba1FRA1U3qHV8oX/bBZ08HIBGg8hQOeZ5I7d5HyzR0971W6KVgDF+s6mRN7d+heNi3XmikbJrozLoEiVQNHIsXjUoc655c2y9NR9Lf5FBweSSrbE34jCUqTA3XmZOdbkjY+ngOcDIfNixRG0yZ57p6UqYW0I+Mg68CB7N+Lv4gFvH/968aML7ConABPGs+vnLdNSQbjuibnaoZwzeSgPoaBJEqBCgNkwO8TyaC04okMj2X7/FGxgZNhwF0V5SVpBllWlGqdAigEF0dher88PbzSIFSm/x8PeACSZWkU0QWV Masum@MASUM-PC", "key_options": null, "keyfile": "/home/sbillah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sbillah", "validate_certs": true} 2019-10-17T20:40:58.908 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:58.909 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-17T20:40:58.952 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:58.953 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-10-17T20:40:59.029 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:59.031 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-10-17T20:40:59.115 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:59.116 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-10-17T20:40:59.128 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.140 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.141 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.153 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:59.153 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-17T20:40:59.175 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-10-17T20:40:59.177 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.189 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.200 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.201 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.213 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.214 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.224 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.225 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.236 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.247 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.248 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.259 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.270 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.270 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:59.272 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.283 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.283 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n', u'name': u'jj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jj", "ovpn": "jj@aurora PFzIfG7NjrCYxJvxd//soA 395884ddf2caa0043f97e4f98530cc3ad646d6f269802a2b79ca1ea7278ee006\njj@metatron iQogIw/KQtewT7oj9Mkivw 0881e5ceb5897e8a370bacee69ad58eb5098090ea4b0d53972214ea7c751e35a\njj@laptop O1e31whZbQ0S7MUtglCRLg 96e39257989ce36e240b5d368e0308d38009d3d923ec398dc9cc6eba371acaa4\njj@aurora2 EtAvlrozxiL3PLYp6mvATg 1018928736c33ed06246f208cd02aa10c0a6efa5b4e34e32408d7a6c72c32e11\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.294 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.295 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.306 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.317 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.318 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.331 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.340 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.340 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.351 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.352 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9', u'name': u'abhishekvrshny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhishekvrshny", "ovpn": "abhishekvrshny@flipkart QTqbWHaqvXwB+yBy6CVO7A 25d026c49dc49b3a1f445d2dc0099d5ed916645b0adb8d0306269ace7a2096e9"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.368 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.368 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.370 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.380 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246', u'name': u'shehbazj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shehbazj", "ovpn": "shehbazj@smrhost oEliXj/jPmfXv9AUAFKB5A 8bb3a682b4ff15de655c0fe610d350c5805d0a970471e4810b648f47e2811246"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.381 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.400 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3', u'name': u'fche'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "fche", "ovpn": "fche@redhat.com LiziWkWg2uWoEHb3Ln92dQ 9c5497793758b069adbba9284dd55944276ba4dac0bb95d9357c81b58174a3c3"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4', u'name': u'onyb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "onyb", "ovpn": "ani@orchid AgbLf9Dufuji5r+9WxM69g b73b7eacb9b628387a17cb1e0a84ff19c29d45dca8f0768e407aa599bc6996c4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.408 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3', u'name': u'oprypin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oprypin", "ovpn": "oprypin@bx-pc-arch 5WYD7PWKwdSQjfGo95ehWw f3c9f170d74c6c443cf0f82f0d87045e1d3a0dbcd01474f6e667ea20a00978b3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.416 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:59.417 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-10-17T20:40:59.418 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.420 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.430 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.430 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.442 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051', u'name': u'asheplyakov'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "asheplyakov", "ovpn": "asheplyakov@asheplyakov.srt.mirantis.net wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.453 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.454 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.464 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.465 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.476 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.477 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0', u'name': u'jwilliamson'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jwilliamson", "ovpn": "jwilliamson@glaurung +d028NM7xCxkVdxxO+b1Lw fece65125073fdc287af724ee4724ad84d2864e758d50dcb23c07b05c3595fe0"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.488 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.500 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.511 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.511 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.522 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.523 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77', u'name': u'liupan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liupan", "ovpn": "liupan@ceph-dev-2 9+Is4mIZgNkYyJLwHvSNOA 5a8fafc187d52041daf4365125692d4619fc557b75560913130c0596f83bbb77"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.534 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:59.535 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6', u'name': u'amarangone'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amarangone", "ovpn": "amarangone@macair.local 9Fslt44BqONCYNhf+uhcnQ 12d46ec6815378a12abc5df00e65235ccbc06ffb0fe5d1db75540a4805cb58b6"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.546 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.547 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57', u'name': u'kmroz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kmroz", "ovpn": "kmroz@suse /JbrIs2mKL5exdmcDnhRgg db4d19ab99c7174429d5ae7b6ca3cf4e04e9bf7810e1826d90f4627643628d57"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.559 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.560 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.571 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.571 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.582 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.583 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.595 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n', u'name': u'adamyanova'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adamyanova", "ovpn": "adamyanova@centos k2nfk8hmGW77nys3R/JkOg 7695b95c2f157d60622e0b0b7ab06fa2cb18661a190d839f7ea587bc44aa0e3c\nadamyanova@ubuntu FmlKgjVzPUxfNDnHeU9vLQ ef7d5524863dfa0787fc5e249873c1a5ea58e7fd5aee27e1d1d33d6f87388a2d\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.596 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.605 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.617 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.628 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.629 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.641 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.652 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.664 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi092.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-17T20:40:59.665 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.680 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha junlGKNU/xzt4OIaGHKBLA e8c67fd935fca490af3fe17453ccae3176268c4bfe1db4a2a879a2ab7ea6bfa5"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.689 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.690 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.701 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.702 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@asus hHsL1xugca0LzY52gKqfqQ 312e0f2680f72d9459c707fcd0ccfb777617f00017f0511839e9b7e3167d590f"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.713 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.724 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.725 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.726 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.736 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.741 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.750 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.751 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.762 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.773 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.774 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.785 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.786 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.825 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi092.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-17T20:40:59.866 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:59 +0000 (0:00:01.586) 0:00:46.304 ****** 2019-10-17T20:40:59.893 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-10-17T20:40:59.895 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:40:59.954 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:40:59 +0000 (0:00:00.087) 0:00:46.391 ****** 2019-10-17T20:40:59.959 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:40:59.960 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-17T20:41:00.068 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-17T20:41:00.132 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-10-17T20:41:00.174 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-17T20:41:00.258 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-10-17T20:41:00.302 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-17T20:41:00.397 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-10-17T20:41:00.419 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-17T20:41:00.420 INFO:teuthology.task.ansible.out: 2019-10-17T20:41:00.534 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-10-17T20:41:00.546 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-17T20:41:00.661 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-10-17T20:41:00.674 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-17T20:41:00.803 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-10-17T20:41:00.806 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-17T20:41:00.932 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-17T20:41:00.933 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-10-17T20:41:01.062 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-17T20:41:01.073 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-10-17T20:41:01.189 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-17T20:41:01.191 INFO:teuthology.task.ansible.out: 2019-10-17T20:41:01.252 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:01.297) 0:00:47.689 ****** 2019-10-17T20:41:01.260 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-10-17T20:41:01.285 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-17T20:41:01.287 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => { "msg": "Host smithi092.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-10-17T20:41:01.331 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.079) 0:00:47.769 ****** 2019-10-17T20:41:01.377 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi092.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-17T20:41:01.419 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.087) 0:00:47.856 ****** 2019-10-17T20:41:01.425 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-10-17T20:41:01.437 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.486 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.067) 0:00:47.924 ****** 2019-10-17T20:41:01.516 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:01.517 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.559 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.073) 0:00:47.997 ****** 2019-10-17T20:41:01.587 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-10-17T20:41:01.589 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.630 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.071) 0:00:48.068 ****** 2019-10-17T20:41:01.642 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:01.643 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-10-17T20:41:01.667 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.702 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.071) 0:00:48.139 ****** 2019-10-17T20:41:01.728 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-17T20:41:01.729 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.766 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.064) 0:00:48.204 ****** 2019-10-17T20:41:01.780 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:01.781 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-10-17T20:41:01.792 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.830 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.064) 0:00:48.268 ****** 2019-10-17T20:41:01.856 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-17T20:41:01.857 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.895 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.064) 0:00:48.332 ****** 2019-10-17T20:41:01.921 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:01.922 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} TASK [common : Set entitlements_path] ****************************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:01.964 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:01 +0000 (0:00:00.069) 0:00:48.402 ****** 2019-10-17T20:41:02.001 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-17T20:41:02.005 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:02.042 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.076) 0:00:48.478 ****** 2019-10-17T20:41:02.046 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:02.047 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-10-17T20:41:02.069 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-17T20:41:02.070 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.108 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.067) 0:00:48.546 ****** 2019-10-17T20:41:02.133 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-17T20:41:02.135 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.183 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.074) 0:00:48.621 ****** 2019-10-17T20:41:02.189 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:02.190 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-10-17T20:41:02.201 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-17T20:41:02.202 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:02.239 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.056) 0:00:48.677 ****** 2019-10-17T20:41:02.265 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-17T20:41:02.267 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.306 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.065) 0:00:48.742 ****** 2019-10-17T20:41:02.311 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:02.312 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-10-17T20:41:02.339 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:02.373 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.068) 0:00:48.811 ****** 2019-10-17T20:41:02.404 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.443 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.064) 0:00:48.875 ****** 2019-10-17T20:41:02.454 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:02.454 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-10-17T20:41:02.468 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.509 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.066) 0:00:48.942 ****** 2019-10-17T20:41:02.530 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-17T20:41:02.532 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.569 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.064) 0:00:49.007 ****** 2019-10-17T20:41:02.584 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:02.585 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-10-17T20:41:02.596 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-17T20:41:02.597 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.634 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.065) 0:00:49.072 ****** 2019-10-17T20:41:02.660 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-17T20:41:02.661 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:02.699 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.064) 0:00:49.136 ****** 2019-10-17T20:41:02.725 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-17T20:41:02.726 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-10-17T20:41:02.767 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.068) 0:00:49.205 ****** 2019-10-17T20:41:02.793 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-17T20:41:02.794 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.831 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.063) 0:00:49.269 ****** 2019-10-17T20:41:02.857 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-17T20:41:02.858 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.896 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.064) 0:00:49.333 ****** 2019-10-17T20:41:02.901 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:02.902 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-10-17T20:41:02.914 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:02.952 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:02 +0000 (0:00:00.056) 0:00:49.389 ****** 2019-10-17T20:41:02.978 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-17T20:41:02.979 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.017 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.065) 0:00:49.455 ****** 2019-10-17T20:41:03.025 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-10-17T20:41:03.036 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-17T20:41:03.037 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.074 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.056) 0:00:49.511 ****** 2019-10-17T20:41:03.101 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:03.152 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.078) 0:00:49.589 ****** 2019-10-17T20:41:03.159 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:03.160 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-10-17T20:41:03.171 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-17T20:41:03.219 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.066) 0:00:49.656 ****** 2019-10-17T20:41:03.247 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-17T20:41:03.248 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.289 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.070) 0:00:49.727 ****** 2019-10-17T20:41:03.294 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:03.295 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-10-17T20:41:03.317 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-17T20:41:03.319 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.360 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.070) 0:00:49.797 ****** 2019-10-17T20:41:03.399 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-17T20:41:03.400 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** ok: [smithi204.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-10-17T20:41:03.402 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-10-17T20:41:03.404 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.448 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.087) 0:00:49.885 ****** 2019-10-17T20:41:03.485 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-17T20:41:03.487 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.497 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.539 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.091) 0:00:49.977 ****** 2019-10-17T20:41:03.570 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:03.571 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-10-17T20:41:03.582 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-17T20:41:03.583 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.623 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.083) 0:00:50.061 ****** 2019-10-17T20:41:03.682 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:03.684 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-10-17T20:41:03.766 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-17T20:41:03.768 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.007870", "end": "2019-10-17 20:41:03.743986", "rc": 0, "start": "2019-10-17 20:41:03.736116", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-17T20:41:03.806 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.183) 0:00:50.244 ****** 2019-10-17T20:41:03.824 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-10-17T20:41:03.825 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-10-17T20:41:03.841 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-17T20:41:03.842 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.880 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.073) 0:00:50.318 ****** 2019-10-17T20:41:03.901 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:03.921 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.041) 0:00:50.359 ****** 2019-10-17T20:41:03.926 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-10-17T20:41:03.927 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:03.968 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:03 +0000 (0:00:00.046) 0:00:50.405 ****** 2019-10-17T20:41:03.973 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-10-17T20:41:03.974 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => { "msg": "Host smithi204.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2019-10-17T20:41:04.017 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.049) 0:00:50.455 ****** 2019-10-17T20:41:04.023 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.065 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.048) 0:00:50.503 ****** 2019-10-17T20:41:04.071 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-10-17T20:41:04.071 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-10-17T20:41:04.113 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.047) 0:00:50.550 ****** 2019-10-17T20:41:04.118 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-17T20:41:04.119 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.159 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.046) 0:00:50.597 ****** 2019-10-17T20:41:04.165 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-10-17T20:41:04.167 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.207 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.047) 0:00:50.644 ****** 2019-10-17T20:41:04.212 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-17T20:41:04.214 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.255 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.048) 0:00:50.693 ****** 2019-10-17T20:41:04.260 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-10-17T20:41:04.261 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.302 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.046) 0:00:50.739 ****** 2019-10-17T20:41:04.307 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-17T20:41:04.307 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.350 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.048) 0:00:50.788 ****** 2019-10-17T20:41:04.365 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.403 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.052) 0:00:50.841 ****** 2019-10-17T20:41:04.408 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-17T20:41:04.410 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.453 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.050) 0:00:50.891 ****** 2019-10-17T20:41:04.458 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-10-17T20:41:04.460 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.506 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.052) 0:00:50.944 ****** 2019-10-17T20:41:04.533 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-10-17T20:41:04.534 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.572 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.065) 0:00:51.010 ****** 2019-10-17T20:41:04.609 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-10-17T20:41:04.610 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.664 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.091) 0:00:51.102 ****** 2019-10-17T20:41:04.691 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-10-17T20:41:04.693 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.737 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.072) 0:00:51.174 ****** 2019-10-17T20:41:04.763 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-10-17T20:41:04.765 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.810 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.072) 0:00:51.247 ****** 2019-10-17T20:41:04.848 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-10-17T20:41:04.849 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:04.850 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:04.891 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.081) 0:00:51.328 ****** 2019-10-17T20:41:04.928 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-10-17T20:41:04.929 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:04.978 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:04 +0000 (0:00:00.086) 0:00:51.415 ****** 2019-10-17T20:41:05.006 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-10-17T20:41:05.007 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.079 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.101) 0:00:51.517 ****** 2019-10-17T20:41:05.109 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-10-17T20:41:05.111 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:05.164 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.084) 0:00:51.601 ****** 2019-10-17T20:41:05.192 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-10-17T20:41:05.194 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.235 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.070) 0:00:51.672 ****** 2019-10-17T20:41:05.263 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:05.307 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.071) 0:00:51.744 ****** 2019-10-17T20:41:05.334 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-10-17T20:41:05.336 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.377 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.070) 0:00:51.815 ****** 2019-10-17T20:41:05.405 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-10-17T20:41:05.406 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.477 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.099) 0:00:51.914 ****** 2019-10-17T20:41:05.505 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-10-17T20:41:05.507 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.551 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.073) 0:00:51.988 ****** 2019-10-17T20:41:05.555 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-10-17T20:41:05.556 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.955643", "end": "2019-10-17 20:41:05.508127", "rc": 0, "start": "2019-10-17 20:41:04.552484", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:05.598 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.047) 0:00:52.035 ****** 2019-10-17T20:41:05.603 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-10-17T20:41:05.604 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.643 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.045) 0:00:52.081 ****** 2019-10-17T20:41:05.648 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-10-17T20:41:05.650 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.692 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.048) 0:00:52.129 ****** 2019-10-17T20:41:05.698 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-10-17T20:41:05.700 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:05.744 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.052) 0:00:52.182 ****** 2019-10-17T20:41:05.780 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-17T20:41:05.782 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.832 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.086) 0:00:52.268 ****** 2019-10-17T20:41:05.857 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-10-17T20:41:05.858 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:05.902 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.071) 0:00:52.339 ****** 2019-10-17T20:41:05.907 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-17T20:41:05.908 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-10-17T20:41:05.910 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.043071", "end": "2019-10-17 20:41:05.844629", "rc": 0, "start": "2019-10-17 20:41:05.801558", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:05.960 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:05 +0000 (0:00:00.058) 0:00:52.398 ****** 2019-10-17T20:41:06.144 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-10-17T20:41:06.145 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.188 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.222) 0:00:52.620 ****** 2019-10-17T20:41:06.209 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-10-17T20:41:06.210 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.248 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.065) 0:00:52.686 ****** 2019-10-17T20:41:06.274 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-10-17T20:41:06.276 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.317 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.069) 0:00:52.755 ****** 2019-10-17T20:41:06.344 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-10-17T20:41:06.346 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.388 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.070) 0:00:52.825 ****** 2019-10-17T20:41:06.414 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-10-17T20:41:06.414 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:06.457 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.069) 0:00:52.895 ****** 2019-10-17T20:41:06.499 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-10-17T20:41:06.541 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.084) 0:00:52.979 ****** 2019-10-17T20:41:06.568 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-10-17T20:41:06.570 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.611 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.069) 0:00:53.048 ****** 2019-10-17T20:41:06.639 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-10-17T20:41:06.640 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.683 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.071) 0:00:53.120 ****** 2019-10-17T20:41:06.734 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.775 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.091) 0:00:53.212 ****** 2019-10-17T20:41:06.776 INFO:teuthology.task.ansible.out: 2019-10-17T20:41:06.813 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-10-17T20:41:06.814 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.856 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.080) 0:00:53.293 ****** 2019-10-17T20:41:06.882 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-10-17T20:41:06.884 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:06.927 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:06 +0000 (0:00:00.070) 0:00:53.364 ****** 2019-10-17T20:41:07.080 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-17T20:41:07.082 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012760", "end": "2019-10-17 20:41:07.235085", "rc": 0, "start": "2019-10-17 20:41:07.222325", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-10-17T20:41:07.128 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.201) 0:00:53.566 ****** 2019-10-17T20:41:07.166 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-10-17T20:41:07.167 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.211 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.082) 0:00:53.649 ****** 2019-10-17T20:41:07.232 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:07.246 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-10-17T20:41:07.248 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.289 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.077) 0:00:53.726 ****** 2019-10-17T20:41:07.315 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-10-17T20:41:07.317 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.367 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.078) 0:00:53.805 ****** 2019-10-17T20:41:07.394 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-10-17T20:41:07.396 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.447 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.079) 0:00:53.885 ****** 2019-10-17T20:41:07.475 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-10-17T20:41:07.477 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.535 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.087) 0:00:53.972 ****** 2019-10-17T20:41:07.571 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-10-17T20:41:07.573 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.620 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.085) 0:00:54.058 ****** 2019-10-17T20:41:07.655 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-10-17T20:41:07.657 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:07.695 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:07 +0000 (0:00:00.074) 0:00:54.133 ****** 2019-10-17T20:41:08.340 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-10-17T20:41:08.341 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.507372", "end": "2019-10-17 20:41:08.500372", "rc": 0, "start": "2019-10-17 20:41:07.993000", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:08.390 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:08 +0000 (0:00:00.694) 0:00:54.827 ****** 2019-10-17T20:41:08.418 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:08.464 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:08 +0000 (0:00:00.073) 0:00:54.901 ****** 2019-10-17T20:41:08.643 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-10-17T20:41:08.644 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.045908", "end": "2019-10-17 20:41:08.806302", "rc": 0, "start": "2019-10-17 20:41:08.760394", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:08.699 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:08 +0000 (0:00:00.235) 0:00:55.137 ****** 2019-10-17T20:41:09.537 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:09.545 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** [WARNING]: Could not find aptitude. Using apt-get instead ok: [smithi092.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:41:09.591 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:09 +0000 (0:00:00.891) 0:00:56.028 ****** 2019-10-17T20:41:10.380 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:41:10.439 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:10 +0000 (0:00:00.847) 0:00:56.876 ****** 2019-10-17T20:41:10.471 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:10.506 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:10 +0000 (0:00:00.066) 0:00:56.942 ****** 2019-10-17T20:41:10.973 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-10-17T20:41:10.973 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-10-17T20:41:11.020 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:11 +0000 (0:00:00.515) 0:00:57.458 ****** 2019-10-17T20:41:11.881 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-10-17T20:41:11.882 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:41:11.925 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:11 +0000 (0:00:00.905) 0:00:58.363 ****** 2019-10-17T20:41:12.120 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:12.126 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi204.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:41:12.169 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:12 +0000 (0:00:00.243) 0:00:58.607 ****** 2019-10-17T20:41:12.244 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi092.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-10-17T20:41:12.549 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-10-17T20:41:12.844 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-10-17T20:41:12.899 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:12 +0000 (0:00:00.730) 0:00:59.337 ****** 2019-10-17T20:41:12.971 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-10-17T20:41:12.973 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:41:13.026 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:13 +0000 (0:00:00.127) 0:00:59.464 ****** 2019-10-17T20:41:13.036 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-17T20:41:13.038 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-10-17T20:41:13.092 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:13 +0000 (0:00:00.065) 0:00:59.529 ****** 2019-10-17T20:41:13.099 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:13.143 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:13 +0000 (0:00:00.051) 0:00:59.580 ****** 2019-10-17T20:41:13.439 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-17T20:41:13.440 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-10-17T20:41:13.462 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-10-17T20:41:13.463 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2019-10-17T20:41:13.503 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:13 +0000 (0:00:00.360) 0:00:59.940 ****** 2019-10-17T20:41:13.763 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-17T20:41:13.764 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-10-17T20:41:14.050 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-10-17T20:41:14.111 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.608) 0:01:00.549 ****** 2019-10-17T20:41:14.151 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-17T20:41:14.153 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-10-17T20:41:14.197 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.086) 0:01:00.635 ****** 2019-10-17T20:41:14.224 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-17T20:41:14.225 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:14.268 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.071) 0:01:00.706 ****** 2019-10-17T20:41:14.307 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-17T20:41:14.309 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:14.366 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.097) 0:01:00.803 ****** 2019-10-17T20:41:14.424 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-17T20:41:14.433 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:14.442 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-10-17T20:41:14.443 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:41:14.444 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi092.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:14.485 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.119) 0:01:00.923 ****** 2019-10-17T20:41:14.525 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.038) 0:01:00.961 ****** 2019-10-17T20:41:14.530 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-17T20:41:14.531 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-nrpe-server", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:14.532 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-basic", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:14.573 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:14 +0000 (0:00:00.048) 0:01:01.010 ****** 2019-10-17T20:41:14.944 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-17T20:41:14.944 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2019-10-17T20:41:15.256 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2019-10-17T20:41:15.432 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-17T20:41:15.434 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "cache_update_time": 1571344592, "cache_updated": false, "changed": false, "item": "nagios-nrpe-server", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:41:15.550 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-10-17T20:41:15.551 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2019-10-17T20:41:15.609 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:15 +0000 (0:00:01.036) 0:01:02.046 ****** 2019-10-17T20:41:15.610 INFO:teuthology.task.ansible.out: 2019-10-17T20:41:15.740 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-10-17T20:41:15.741 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2019-10-17T20:41:15.782 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:15 +0000 (0:00:00.173) 0:01:02.219 ****** 2019-10-17T20:41:16.101 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-17T20:41:16.101 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2019-10-17T20:41:16.191 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-17T20:41:16.193 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "cache_update_time": 1571344592, "cache_updated": false, "changed": false, "item": "nagios-plugins-basic", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:41:16.235 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.452) 0:01:02.672 ****** 2019-10-17T20:41:16.387 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-17T20:41:16.388 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2019-10-17T20:41:16.547 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-10-17T20:41:16.548 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-10-17T20:41:16.590 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.355) 0:01:03.028 ****** 2019-10-17T20:41:16.670 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-10-17T20:41:16.672 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2019-10-17T20:41:16.713 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.122) 0:01:03.150 ****** 2019-10-17T20:41:16.751 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-10-17T20:41:16.753 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2019-10-17T20:41:16.792 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.079) 0:01:03.230 ****** 2019-10-17T20:41:16.817 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-10-17T20:41:16.818 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:16.862 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.069) 0:01:03.299 ****** 2019-10-17T20:41:16.889 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-10-17T20:41:16.891 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:16.933 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.070) 0:01:03.370 ****** 2019-10-17T20:41:16.952 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-17T20:41:16.953 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-10-17T20:41:17.000 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:16 +0000 (0:00:00.067) 0:01:03.438 ****** 2019-10-17T20:41:17.005 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-10-17T20:41:17.006 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:17.007 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:17.009 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:17.054 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.053) 0:01:03.492 ****** 2019-10-17T20:41:17.111 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-10-17T20:41:17.113 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-nrpe-server", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "changed": false, "item": "nagios-plugins-basic", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:17.166 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.111) 0:01:03.603 ****** 2019-10-17T20:41:17.173 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-17T20:41:17.174 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:41:17.220 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.053) 0:01:03.658 ****** 2019-10-17T20:41:17.260 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-10-17T20:41:17.262 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:17.316 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.095) 0:01:03.753 ****** 2019-10-17T20:41:17.344 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-17T20:41:17.345 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:17.385 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.069) 0:01:03.823 ****** 2019-10-17T20:41:17.412 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-17T20:41:17.414 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:17.456 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.070) 0:01:03.894 ****** 2019-10-17T20:41:17.788 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-10-17T20:41:17.822 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:17 +0000 (0:00:00.365) 0:01:04.259 ****** 2019-10-17T20:41:18.091 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-17T20:41:18.093 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nagios-nrpe-server) => {"ansible_loop_var": "item", "cache_update_time": 1571344640, "cache_updated": false, "changed": false, "item": "nagios-nrpe-server", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:41:18.437 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-10-17T20:41:18.439 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-10-17 20:40:06 UTC", "ActiveEnterTimestampMonotonic": "18599281", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket local-fs.target -.mount nss-lookup.target network.target var-run.mount remote-fs.target sysinit.target systemd-tmpfiles-setup.service time-sync.target system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-10-17 20:40:06 UTC", "AssertTimestampMonotonic": "18598738", "Before": "getty@tty1.service shutdown.target plymouth-quit.service multi-user.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-10-17 20:40:06 UTC", "ConditionTimestampMonotonic": "18598738", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-10-17T20:41:18.440 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "815", "ExecMainStartTimestamp": "Thu 2019-10-17 20:40:06 UTC", "ExecMainStartTimestampMonotonic": "18599224", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2019-10-17 20:40:06 UTC] ; stop_time=[n/a] ; pid=815 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-10-17 20:40:06 UTC", "InactiveExitTimestampMonotonic": "18599281", "InvocationID": "a88641b0e5e04d838431bfa349ce9389", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "inf 2019-10-17T20:41:18.442 INFO:teuthology.task.ansible.out:inity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "815", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "R 2019-10-17T20:41:18.443 INFO:teuthology.task.ansible.out:estrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-10-17 20:40:06 UTC", "StateChangeTimestampMonotonic": "18599281", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-10-17 20:40:06 UTC", "WatchdogTimestampMonotonic": "18599281", "WatchdogUSec": "0"}} 2019-10-17T20:41:18.483 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.661) 0:01:04.921 ****** 2019-10-17T20:41:18.513 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-17T20:41:18.517 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.552 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.068) 0:01:04.990 ****** 2019-10-17T20:41:18.589 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-17T20:41:18.591 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.636 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.083) 0:01:05.073 ****** 2019-10-17T20:41:18.663 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-17T20:41:18.664 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.705 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.069) 0:01:05.143 ****** 2019-10-17T20:41:18.733 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-17T20:41:18.734 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.781 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.075) 0:01:05.218 ****** 2019-10-17T20:41:18.809 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.848 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.067) 0:01:05.286 ****** 2019-10-17T20:41:18.875 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-17T20:41:18.877 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.920 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.072) 0:01:05.358 ****** 2019-10-17T20:41:18.947 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-10-17T20:41:18.948 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:18.990 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:18 +0000 (0:00:00.069) 0:01:05.427 ****** 2019-10-17T20:41:18.996 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-10-17T20:41:18.996 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nagios-plugins-basic) => {"ansible_loop_var": "item", "cache_update_time": 1571344640, "cache_updated": false, "changed": false, "item": "nagios-plugins-basic", "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:41:19.039 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.049) 0:01:05.477 ****** 2019-10-17T20:41:19.044 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-17T20:41:19.045 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:19.089 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.049) 0:01:05.526 ****** 2019-10-17T20:41:19.114 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-17T20:41:19.116 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:19.161 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.072) 0:01:05.599 ****** 2019-10-17T20:41:19.188 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-17T20:41:19.189 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:19.233 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.071) 0:01:05.670 ****** 2019-10-17T20:41:19.262 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-17T20:41:19.263 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-10-17T20:41:19.303 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.069) 0:01:05.740 ****** 2019-10-17T20:41:19.343 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-17T20:41:19.345 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-10-17T20:41:19.347 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-10-17T20:41:19.349 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2019-10-17T20:41:19.411 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.106) 0:01:05.847 ****** 2019-10-17T20:41:19.447 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.037) 0:01:05.885 ****** 2019-10-17T20:41:19.452 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-17T20:41:19.452 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-10-17T20:41:19.497 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.049) 0:01:05.934 ****** 2019-10-17T20:41:19.535 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-17T20:41:19.535 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-17T20:41:19.585 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.088) 0:01:06.023 ****** 2019-10-17T20:41:19.716 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-17T20:41:19.718 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-17T20:41:19.760 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.175) 0:01:06.198 ****** 2019-10-17T20:41:19.841 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-10-17T20:41:19.842 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2019-10-17T20:41:19.901 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.140) 0:01:06.338 ****** 2019-10-17T20:41:19.906 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-17T20:41:19.907 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2019-10-17T20:41:19.952 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:19 +0000 (0:00:00.050) 0:01:06.389 ****** 2019-10-17T20:41:19.989 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-17T20:41:19.993 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:20.056 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.102) 0:01:06.492 ****** 2019-10-17T20:41:20.064 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-10-17T20:41:20.065 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:41:20.076 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:20.109 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.054) 0:01:06.546 ****** 2019-10-17T20:41:20.114 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-10-17T20:41:20.116 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:20.167 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.057) 0:01:06.604 ****** 2019-10-17T20:41:20.173 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-10-17T20:41:20.175 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:20.219 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.052) 0:01:06.656 ****** 2019-10-17T20:41:20.245 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-10-17T20:41:20.247 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-10-17T20:41:20.287 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.068) 0:01:06.725 ****** 2019-10-17T20:41:20.315 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-10-17T20:41:20.316 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:20.318 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi092.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-10-17T20:41:20.358 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.070) 0:01:06.795 ****** 2019-10-17T20:41:20.398 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.039) 0:01:06.835 ****** 2019-10-17T20:41:20.731 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2019-10-17T20:41:20.772 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.374) 0:01:07.210 ****** 2019-10-17T20:41:20.801 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-10-17T20:41:20.803 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-10-17T20:41:20.865 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.092) 0:01:07.303 ****** 2019-10-17T20:41:20.905 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-17T20:41:20.908 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi092"}, "changed": false} 2019-10-17T20:41:20.945 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:20 +0000 (0:00:00.079) 0:01:07.382 ****** 2019-10-17T20:41:20.989 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-10-17T20:41:20.992 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-10-17 20:39:59 UTC", "ActiveEnterTimestampMonotonic": "16889994", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target -.mount local-fs.target remote-fs.target nss-lookup.target system.slice systemd-tmpfiles-setup.service var-run.mount sysinit.target time-sync.target basic.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-10-17 20:39:59 UTC", "AssertTimestampMonotonic": "16889417", "Before": "getty@tty1.service xdm.service plymouth-quit.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-10-17 20:39:59 UTC", "ConditionTimestampMonotonic": "16889417", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2019-10-17T20:41:20.993 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "807", "ExecMainStartTimestamp": "Thu 2019-10-17 20:39:59 UTC", "ExecMainStartTimestampMonotonic": "16889966", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2019-10-17 20:39:59 UTC] ; stop_time=[n/a] ; pid=807 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-10-17 20:39:59 UTC", "InactiveExitTimestampMonotonic": "16889994", "InvocationID": "d4dec80eb190440099786943b4e72496", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "inf 2019-10-17T20:41:20.994 INFO:teuthology.task.ansible.out:inity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "807", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "R 2019-10-17T20:41:20.996 INFO:teuthology.task.ansible.out:estrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-10-17 20:39:59 UTC", "StateChangeTimestampMonotonic": "16889994", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-10-17 20:39:59 UTC", "WatchdogTimestampMonotonic": "16889993", "WatchdogUSec": "0"}} 2019-10-17T20:41:21.043 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.098) 0:01:07.481 ****** 2019-10-17T20:41:21.071 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-10-17T20:41:21.073 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.115 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.071) 0:01:07.552 ****** 2019-10-17T20:41:21.152 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-10-17T20:41:21.154 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=libsemanage-python) => {"ansible_loop_var": "item", "changed": false, "item": "libsemanage-python", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=policycoreutils-python) => {"ansible_loop_var": "item", "changed": false, "item": "policycoreutils-python", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.209 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.094) 0:01:07.647 ****** 2019-10-17T20:41:21.237 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-10-17T20:41:21.239 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.304 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.094) 0:01:07.741 ****** 2019-10-17T20:41:21.332 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-10-17T20:41:21.333 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.373 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.069) 0:01:07.811 ****** 2019-10-17T20:41:21.410 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-10-17T20:41:21.412 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.474 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.100) 0:01:07.911 ****** 2019-10-17T20:41:21.506 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-10-17T20:41:21.508 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.547 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.071) 0:01:07.983 ****** 2019-10-17T20:41:21.572 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-10-17T20:41:21.574 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.613 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.067) 0:01:08.051 ****** 2019-10-17T20:41:21.654 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-10-17T20:41:21.655 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.698 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.084) 0:01:08.135 ****** 2019-10-17T20:41:21.725 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-10-17T20:41:21.727 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.765 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.067) 0:01:08.203 ****** 2019-10-17T20:41:21.801 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-10-17T20:41:21.803 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:21.842 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.077) 0:01:08.280 ****** 2019-10-17T20:41:21.869 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-10-17T20:41:21.869 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2019-10-17T20:41:21.912 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.069) 0:01:08.349 ****** 2019-10-17T20:41:21.928 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-17T20:41:21.928 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi092.front.sepia.ceph.com", "ansible_hostname": "smithi092", "ansible_nodename": "smithi092"}, "changed": true, "name": "smithi092"} 2019-10-17T20:41:21.969 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:21 +0000 (0:00:00.057) 0:01:08.407 ****** 2019-10-17T20:41:21.975 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-10-17T20:41:21.977 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2019-10-17T20:41:22.019 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.049) 0:01:08.457 ****** 2019-10-17T20:41:22.056 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-10-17T20:41:22.057 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2019-10-17T20:41:22.097 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.077) 0:01:08.535 ****** 2019-10-17T20:41:22.135 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-10-17T20:41:22.137 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-10-17T20:41:22.176 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.079) 0:01:08.614 ****** 2019-10-17T20:41:22.298 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-10-17T20:41:22.299 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-10-17T20:41:22.345 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.168) 0:01:08.783 ****** 2019-10-17T20:41:22.351 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-10-17T20:41:22.352 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2019-10-17T20:41:22.397 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.052) 0:01:08.835 ****** 2019-10-17T20:41:22.541 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-10-17T20:41:22.543 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "name": "ubuntu", "state": "present", "system": false} 2019-10-17T20:41:22.594 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.196) 0:01:09.031 ****** 2019-10-17T20:41:22.620 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-10-17T20:41:22.622 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:22.670 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.076) 0:01:09.108 ****** 2019-10-17T20:41:22.678 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-10-17T20:41:22.679 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-10-17T20:41:22.686 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:22.725 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.055) 0:01:09.163 ****** 2019-10-17T20:41:22.731 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-10-17T20:41:22.732 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:22.778 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:22 +0000 (0:00:00.052) 0:01:09.215 ****** 2019-10-17T20:41:22.938 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-10-17T20:41:22.939 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2019-10-17T20:41:23.015 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.237) 0:01:09.453 ****** 2019-10-17T20:41:23.285 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:23.298 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-17T20:41:23.300 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually changed: [smithi092.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-17T20:41:23.356 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.340) 0:01:09.793 ****** 2019-10-17T20:41:23.383 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-17T20:41:23.385 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.428 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.072) 0:01:09.866 ****** 2019-10-17T20:41:23.436 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-10-17T20:41:23.437 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2019-10-17T20:41:23.489 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.060) 0:01:09.927 ****** 2019-10-17T20:41:23.495 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-17T20:41:23.496 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.542 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.052) 0:01:09.979 ****** 2019-10-17T20:41:23.548 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-10-17T20:41:23.549 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi204"}, "changed": false} 2019-10-17T20:41:23.599 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.056) 0:01:10.036 ****** 2019-10-17T20:41:23.604 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-17T20:41:23.609 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.647 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.048) 0:01:10.084 ****** 2019-10-17T20:41:23.684 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-17T20:41:23.685 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.735 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.088) 0:01:10.173 ****** 2019-10-17T20:41:23.766 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-17T20:41:23.767 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.808 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.072) 0:01:10.246 ****** 2019-10-17T20:41:23.834 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-17T20:41:23.836 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.875 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.067) 0:01:10.313 ****** 2019-10-17T20:41:23.903 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-17T20:41:23.905 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:23.945 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:23 +0000 (0:00:00.069) 0:01:10.383 ****** 2019-10-17T20:41:23.972 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-17T20:41:24.019 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.069) 0:01:10.452 ****** 2019-10-17T20:41:24.052 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-17T20:41:24.095 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.079) 0:01:10.532 ****** 2019-10-17T20:41:24.112 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-10-17T20:41:24.112 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi204.front.sepia.ceph.com", "ansible_hostname": "smithi204", "ansible_nodename": "smithi204"}, "changed": true, "name": "smithi204"} 2019-10-17T20:41:24.154 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.058) 0:01:10.591 ****** 2019-10-17T20:41:24.160 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.205 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.051) 0:01:10.642 ****** 2019-10-17T20:41:24.242 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-17T20:41:24.244 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.254 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.297 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.091) 0:01:10.734 ****** 2019-10-17T20:41:24.324 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-10-17T20:41:24.326 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.367 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.069) 0:01:10.804 ****** 2019-10-17T20:41:24.404 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:41:24.406 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.446 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.079) 0:01:10.883 ****** 2019-10-17T20:41:24.474 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.516 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.068) 0:01:10.952 ****** 2019-10-17T20:41:24.535 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2019-10-17T20:41:24.567 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.052) 0:01:11.004 ****** 2019-10-17T20:41:24.572 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-17T20:41:24.573 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.620 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.052) 0:01:11.057 ****** 2019-10-17T20:41:24.646 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-17T20:41:24.648 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.688 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.068) 0:01:11.125 ****** 2019-10-17T20:41:24.714 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-17T20:41:24.716 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.756 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.068) 0:01:11.194 ****** 2019-10-17T20:41:24.793 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-17T20:41:24.794 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.796 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.816 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "radosgw", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.817 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.828 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.839 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.840 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": false, "item": "librbd1", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.850 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": false, "item": "librados2", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.852 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=ceph-fs-common-dbg) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common-dbg", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.863 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=ceph-fs-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:24.908 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.151) 0:01:11.346 ****** 2019-10-17T20:41:24.914 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-10-17T20:41:24.915 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2019-10-17T20:41:24.956 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.048) 0:01:11.394 ****** 2019-10-17T20:41:24.962 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-17T20:41:25.003 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:24 +0000 (0:00:00.046) 0:01:11.440 ****** 2019-10-17T20:41:25.050 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-17T20:41:25.052 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=qemu-system-x86) => {"ansible_loop_var": "item", "changed": false, "item": "qemu-system-x86", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.070 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=lttng-tools) => {"ansible_loop_var": "item", "changed": false, "item": "lttng-tools", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.071 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=libtool-bin) => {"ansible_loop_var": "item", "changed": false, "item": "libtool-bin", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.116 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.110) 0:01:11.551 ****** 2019-10-17T20:41:25.142 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-17T20:41:25.190 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.076) 0:01:11.627 ****** 2019-10-17T20:41:25.221 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-17T20:41:25.222 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=openmpi-common) => {"ansible_loop_var": "item", "changed": false, "item": "openmpi-common", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.270 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.080) 0:01:11.708 ****** 2019-10-17T20:41:25.308 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-17T20:41:25.310 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=apt) => {"ansible_loop_var": "item", "changed": false, "item": "apt", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=apache2) => {"ansible_loop_var": "item", "changed": false, "item": "apache2", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.357 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.086) 0:01:11.794 ****** 2019-10-17T20:41:25.386 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-17T20:41:25.387 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.431 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.074) 0:01:11.868 ****** 2019-10-17T20:41:25.460 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-10-17T20:41:25.462 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.472 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:25.510 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.078) 0:01:11.947 ****** 2019-10-17T20:41:25.517 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-10-17T20:41:25.518 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-10-17T20:41:25.572 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.062) 0:01:12.009 ****** 2019-10-17T20:41:25.577 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.624 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.052) 0:01:12.062 ****** 2019-10-17T20:41:25.631 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-10-17T20:41:25.631 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.677 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.052) 0:01:12.115 ****** 2019-10-17T20:41:25.703 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-17T20:41:25.704 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.743 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.066) 0:01:12.181 ****** 2019-10-17T20:41:25.759 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-17T20:41:25.760 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004579", "end": "2019-10-17 20:41:25.743345", "rc": 0, "start": "2019-10-17 20:41:25.738766", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:25.801 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.058) 0:01:12.239 ****** 2019-10-17T20:41:25.807 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-10-17T20:41:25.808 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.850 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.048) 0:01:12.287 ****** 2019-10-17T20:41:25.855 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-17T20:41:25.902 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.052) 0:01:12.340 ****** 2019-10-17T20:41:25.908 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-17T20:41:25.910 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:25.951 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:25 +0000 (0:00:00.048) 0:01:12.389 ****** 2019-10-17T20:41:25.988 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-17T20:41:25.990 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.029 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.077) 0:01:12.466 ****** 2019-10-17T20:41:26.066 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-17T20:41:26.067 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.109 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.077) 0:01:12.544 ****** 2019-10-17T20:41:26.134 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-17T20:41:26.135 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.172 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.066) 0:01:12.610 ****** 2019-10-17T20:41:26.199 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-10-17T20:41:26.243 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.069) 0:01:12.680 ****** 2019-10-17T20:41:26.248 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-10-17T20:41:26.249 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-10-17T20:41:26.291 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.048) 0:01:12.728 ****** 2019-10-17T20:41:26.297 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-10-17T20:41:26.345 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.053) 0:01:12.782 ****** 2019-10-17T20:41:26.350 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-10-17T20:41:26.351 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.392 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.047) 0:01:12.829 ****** 2019-10-17T20:41:26.398 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-10-17T20:41:26.400 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.441 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.048) 0:01:12.878 ****** 2019-10-17T20:41:26.500 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-10-17T20:41:26.502 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.545 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.104) 0:01:12.983 ****** 2019-10-17T20:41:26.572 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.617 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.072) 0:01:13.055 ****** 2019-10-17T20:41:26.656 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:41:26.657 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.700 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.082) 0:01:13.138 ****** 2019-10-17T20:41:26.726 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-10-17T20:41:26.727 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.769 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.068) 0:01:13.207 ****** 2019-10-17T20:41:26.795 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-10-17T20:41:26.797 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.837 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.067) 0:01:13.274 ****** 2019-10-17T20:41:26.864 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-10-17T20:41:26.866 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.908 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.071) 0:01:13.346 ****** 2019-10-17T20:41:26.935 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-10-17T20:41:26.937 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:26.997 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:26 +0000 (0:00:00.088) 0:01:13.434 ****** 2019-10-17T20:41:27.034 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-17T20:41:27.036 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=ceph) => {"ansible_loop_var": "item", "changed": false, "item": "ceph", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=ceph-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-common", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.046 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=libcephfs1) => {"ansible_loop_var": "item", "changed": false, "item": "libcephfs1", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.047 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=radosgw) => {"ansible_loop_var": "item", "changed": false, "item": "radosgw", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.057 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-ceph) => {"ansible_loop_var": "item", "changed": false, "item": "python-ceph", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.058 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-rados) => {"ansible_loop_var": "item", "changed": false, "item": "python-rados", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.071 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-cephfs) => {"ansible_loop_var": "item", "changed": false, "item": "python-cephfs", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.071 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=python-rbd) => {"ansible_loop_var": "item", "changed": false, "item": "python-rbd", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.089 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=librbd1) => {"ansible_loop_var": "item", "changed": false, "item": "librbd1", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.090 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=librados2) => {"ansible_loop_var": "item", "changed": false, "item": "librados2", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.091 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=ceph-fs-common-dbg) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common-dbg", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=ceph-fs-common) => {"ansible_loop_var": "item", "changed": false, "item": "ceph-fs-common", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.138 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.141) 0:01:13.576 ****** 2019-10-17T20:41:27.170 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-17T20:41:27.227 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.087) 0:01:13.663 ****** 2019-10-17T20:41:27.263 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-17T20:41:27.264 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=mpich) => {"ansible_loop_var": "item", "changed": false, "item": "mpich", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.266 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=qemu-system-x86) => {"ansible_loop_var": "item", "changed": false, "item": "qemu-system-x86", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=lttng-tools) => {"ansible_loop_var": "item", "changed": false, "item": "lttng-tools", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.276 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=libtool-bin) => {"ansible_loop_var": "item", "changed": false, "item": "libtool-bin", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.321 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.095) 0:01:13.759 ****** 2019-10-17T20:41:27.348 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-10-17T20:41:27.393 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.071) 0:01:13.831 ****** 2019-10-17T20:41:27.438 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi204.front.sepia.ceph.com] => (item=openmpi-common) => {"ansible_loop_var": "item", "changed": false, "item": "openmpi-common", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.476 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.083) 0:01:13.914 ****** 2019-10-17T20:41:27.524 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-17T20:41:27.526 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=apt) => {"ansible_loop_var": "item", "changed": false, "item": "apt", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.536 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=apache2) => {"ansible_loop_var": "item", "changed": false, "item": "apache2", "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.582 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.105) 0:01:14.020 ****** 2019-10-17T20:41:27.610 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-10-17T20:41:27.612 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.650 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.067) 0:01:14.088 ****** 2019-10-17T20:41:27.677 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.716 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.066) 0:01:14.154 ****** 2019-10-17T20:41:27.754 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi092.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-10-17T20:41:27.756 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:27.812 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:27 +0000 (0:00:00.095) 0:01:14.249 ****** 2019-10-17T20:41:27.987 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-10-17T20:41:27.992 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005001", "end": "2019-10-17 20:41:28.130484", "rc": 0, "start": "2019-10-17 20:41:28.125483", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:28.054 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:28 +0000 (0:00:00.242) 0:01:14.492 ****** 2019-10-17T20:41:28.082 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-10-17T20:41:28.135 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:28 +0000 (0:00:00.080) 0:01:14.573 ****** 2019-10-17T20:41:28.479 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2019-10-17T20:41:28.533 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:28 +0000 (0:00:00.398) 0:01:14.971 ****** 2019-10-17T20:41:28.565 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:41:28.605 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:28 +0000 (0:00:00.071) 0:01:15.043 ****** 2019-10-17T20:41:28.727 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-17T20:41:28.728 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-10-17T20:41:28.789 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:28 +0000 (0:00:00.183) 0:01:15.226 ****** 2019-10-17T20:41:29.667 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:41:29.722 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:29 +0000 (0:00:00.933) 0:01:16.160 ****** 2019-10-17T20:41:29.729 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-17T20:41:29.729 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2019-10-17T20:41:29.751 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-17T20:41:29.791 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:29 +0000 (0:00:00.069) 0:01:16.229 ****** 2019-10-17T20:41:29.963 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-17T20:41:29.965 INFO:teuthology.task.ansible.out: [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035887", "end": "2019-10-17 20:41:29.947954", "rc": 0, "start": "2019-10-17 20:41:29.912067", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:30.006 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:30 +0000 (0:00:00.214) 0:01:16.443 ****** 2019-10-17T20:41:30.576 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-10-17T20:41:30.581 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2019-10-17T20:41:30.619 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:30 +0000 (0:00:00.613) 0:01:17.057 ****** 2019-10-17T20:41:31.501 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-10-17T20:41:31.503 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:41:31.546 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:31 +0000 (0:00:00.927) 0:01:17.984 ****** 2019-10-17T20:41:31.572 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-10-17T20:41:31.616 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:31 +0000 (0:00:00.069) 0:01:18.053 ****** 2019-10-17T20:41:31.840 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-10-17T20:41:31.841 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.072699", "end": "2019-10-17 20:41:31.982127", "rc": 0, "start": "2019-10-17 20:41:31.909428", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:41:31.882 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:31 +0000 (0:00:00.265) 0:01:18.319 ****** 2019-10-17T20:41:32.693 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-10-17T20:41:32.694 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:41:32.737 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:32 +0000 (0:00:00.855) 0:01:19.175 ****** 2019-10-17T20:41:32.759 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:34.535 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:41:34.583 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:34 +0000 (0:00:01.845) 0:01:21.021 ****** 2019-10-17T20:41:34.604 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove|list + packages_to_remove|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:43.257 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-10-17T20:41:43.258 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item=[u'ceph', u'ceph-common', u'libcephfs1', u'radosgw', u'python-ceph', u'python-rados', u'python-cephfs', u'python-rbd', u'librbd1', u'librados2', u'ceph-fs-common-dbg', u'ceph-fs-common', u'openmpi-common']) => {"ansible_loop_var": "item", "changed": true, "item": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common", "openmpi-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 95 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading 2019-10-17T20:41:43.260 INFO:teuthology.task.ansible.out:database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136799 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.1) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 2019-10-17T20:41:43.261 INFO:teuthology.task.ansible.out:6 to remove and 95 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136799 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.1) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:41:43.302 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:43 +0000 (0:00:08.718) 0:01:29.740 ****** 2019-10-17T20:41:43.320 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:41:44.424 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item=[u'ceph', u'ceph-common', u'libcephfs1', u'radosgw', u'python-ceph', u'python-rados', u'python-cephfs', u'python-rbd', u'librbd1', u'librados2', u'ceph-fs-common-dbg', u'ceph-fs-common', u'openmpi-common']) => {"ansible_loop_var": "item", "changed": true, "item": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common", "openmpi-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 95 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading 2019-10-17T20:41:44.426 INFO:teuthology.task.ansible.out:database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136799 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.1) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 2019-10-17T20:41:44.427 INFO:teuthology.task.ansible.out:6 to remove and 95 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136799 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.17) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.17) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.1) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:41:44.473 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:41:44 +0000 (0:00:01.171) 0:01:30.911 ****** 2019-10-17T20:41:44.493 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:06.619 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-17T20:42:06.624 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item=[u'apt', u'apache2']) => {"ansible_loop_var": "item", "cache_update_time": 1571344592, "cache_updated": false, "changed": true, "item": ["apt", "apache2"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0\nSuggested packages:\n www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0\n7 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.\nNeed to get 3622 kB of archives.\nAfter this operation, 41.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd6 2019-10-17T20:42:06.627 INFO:teuthology.task.ansible.out:4 1.6.12 [1200 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]\nFetched 3622 kB in 4s (959 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136657 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 13665 2019-10-17T20:42:06.629 INFO:teuthology.task.ansible.out:7 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_1.6.12_amd64.deb ...\r\nUnpacking apt (1.6.12) over (1.6.11) ...\r\nSetting up apt (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136663 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12_amd64.deb ...\r\nUnpacking apt-utils (1.6.12) over (1.6.11) ...\r\nPreparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...\r\nUnpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nProcessing triggers for ufw (0.36-0ubuntu0.18.04.1) ...\r\nProcessing triggers for ureadahead (0.100.0-21) ...\r\nureadahead will be reprofiled on next reboot\r\nSetting up apt-utils (1.6.12) ...\r\nSetting up apache2-utils (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-bin (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-data (2.4.29-1ubuntu4.11) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for systemd (237-3ubuntu10.25) ...\r\nProcessing tri 2019-10-17T20:42:06.630 INFO:teuthology.task.ansible.out:ggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up apache2 (2.4.29-1ubuntu4.11) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0", "Suggested packages:", " www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0", "7 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.", "Need to get 3622 kB of archives.", "After this operation, 41.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12 [1200 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]", "Get:5 http://archive.ubuntu. 2019-10-17T20:42:06.632 INFO:teuthology.task.ansible.out:com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]", "Fetched 3622 kB in 4s (959 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../archives/apt_1.6.12_amd64.deb ...", "Unpacking apt (1.6.12) over (1.6.11) ...", "Setting u 2019-10-17T20:42:06.633 INFO:teuthology.task.ansible.out:p apt (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136663 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12_amd64.deb ...", "Unpacking apt-utils (1.6.12) over (1.6.11) ...", "Preparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...", "Unpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...", "Processing triggers for ureadahead (0.100.0-21) ...", "ureadahead will be reprofiled on next reboot", "Setting up apt-utils (1.6.12) ...", "Setting up apache2-utils (2.4.29-1ubuntu4.11) ...", "Setting up apache2-bin (2.4.29-1ubuntu4.11) ...", "Setting up apache2-data (2.4.29-1ubuntu4.11) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for systemd (237-3ubuntu10.25) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up apache2 (2.4.29-1ubuntu4.11) ..."], "warnings": ["Could not find ap 2019-10-17T20:42:06.635 INFO:teuthology.task.ansible.out:titude. Using apt-get instead"]} 2019-10-17T20:42:06.675 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:06 +0000 (0:00:22.201) 0:01:53.113 ****** 2019-10-17T20:42:06.705 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-17T20:42:06.746 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:06 +0000 (0:00:00.070) 0:01:53.184 ****** 2019-10-17T20:42:06.776 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:06.884 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-17T20:42:06.886 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item=[u'apt', u'apache2']) => {"ansible_loop_var": "item", "cache_update_time": 1571344640, "cache_updated": false, "changed": true, "item": ["apt", "apache2"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0\nSuggested packages:\n www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0\n7 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.\nNeed to get 3622 kB of archives.\nAfter this operation, 41.0 kB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd6 2019-10-17T20:42:06.888 INFO:teuthology.task.ansible.out:4 1.6.12 [1200 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]\nFetched 3622 kB in 4s (976 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136657 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 13665 2019-10-17T20:42:06.889 INFO:teuthology.task.ansible.out:7 files and directories currently installed.)\r\nPreparing to unpack .../archives/apt_1.6.12_amd64.deb ...\r\nUnpacking apt (1.6.12) over (1.6.11) ...\r\nSetting up apt (1.6.12) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136663 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12_amd64.deb ...\r\nUnpacking apt-utils (1.6.12) over (1.6.11) ...\r\nPreparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2 (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...\r\nUnpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nPreparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...\r\nUnpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...\r\nProcessing triggers for ufw (0.36-0ubuntu0.18.04.1) ...\r\nProcessing triggers for ureadahead (0.100.0-21) ...\r\nureadahead will be reprofiled on next reboot\r\nSetting up apt-utils (1.6.12) ...\r\nSetting up apache2-utils (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-bin (2.4.29-1ubuntu4.11) ...\r\nSetting up apache2-data (2.4.29-1ubuntu4.11) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for systemd (237-3ubuntu10.25) ...\r\nProcessing tri 2019-10-17T20:42:06.891 INFO:teuthology.task.ansible.out:ggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up apache2 (2.4.29-1ubuntu4.11) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 2019-10-17T20:42:06.892 INFO:teuthology.task.ansible.out: libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apache2-bin apache2-data apache2-utils apt-utils libapt-pkg5.0", "Suggested packages:", " www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apache2 apache2-bin apache2-data apache2-utils apt apt-utils libapt-pkg5.0", "7 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.", "Need to get 3622 kB of archives.", "After this operation, 41.0 kB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12 [806 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12 [1200 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12 [206 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2 amd64 2.4.29-1ubuntu4.11 [95.1 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-bin amd64 2.4.29-1ubuntu4.11 [1071 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.11 [83.9 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-data all 2.4.29-1ubuntu4.11 [160 kB]", "Fetched 3622 kB in 4s (976 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading dat 2019-10-17T20:42:06.894 INFO:teuthology.task.ansible.out:abase ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12) over (1.6.11) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136657 files and directories currently installed.)", "Preparing to unpack .../archives/apt_1.6.12_amd64.deb ...", "Unpacking apt (1.6.12) over (1.6.11) ...", "Setting up apt (1.6.12) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136663 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12_amd64.deb ...", "Unpacking apt-utils (1.6.12) over (1.6.11) ...", "Preparing to unpack .../apache2_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2 (2.4.29-1ubunt 2019-10-17T20:42:06.895 INFO:teuthology.task.ansible.out:u4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-bin_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-bin (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.11_amd64.deb ...", "Unpacking apache2-utils (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Preparing to unpack .../apache2-data_2.4.29-1ubuntu4.11_all.deb ...", "Unpacking apache2-data (2.4.29-1ubuntu4.11) over (2.4.29-1ubuntu4.8) ...", "Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ...", "Processing triggers for ureadahead (0.100.0-21) ...", "ureadahead will be reprofiled on next reboot", "Setting up apt-utils (1.6.12) ...", "Setting up apache2-utils (2.4.29-1ubuntu4.11) ...", "Setting up apache2-bin (2.4.29-1ubuntu4.11) ...", "Setting up apache2-data (2.4.29-1ubuntu4.11) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for systemd (237-3ubuntu10.25) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up apache2 (2.4.29-1ubuntu4.11) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:06.936 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:06 +0000 (0:00:00.189) 0:01:53.374 ****** 2019-10-17T20:42:06.962 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-10-17T20:42:07.002 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:06 +0000 (0:00:00.065) 0:01:53.440 ****** 2019-10-17T20:42:07.031 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages|list + common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:18.955 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-17T20:42:18.957 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item=[u'mpich', u'qemu-system-x86', u'lttng-tools', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libev-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common']) => {"ansible_loop_var": "item", "cache_update_time": 1571344592, "cache_updated": false, "changed": true, "item": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2019-10-17T20:42:18.958 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 88 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]\nFetched 10.4 MB in 4s (2687 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading 2019-10-17T20:42:18.960 INFO:teuthology.task.ansible.out:database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136665 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest 2019-10-17T20:42:18.962 INFO:teuthology.task.ansible.out:version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 88 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]", "Fetched 10.4 MB in 4s (2687 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136665 files a 2019-10-17T20:42:18.963 INFO:teuthology.task.ansible.out:nd directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:19.023 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:19 +0000 (0:00:12.021) 0:02:05.461 ****** 2019-10-17T20:42:19.040 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item=[u'mpich', u'qemu-system-x86', u'lttng-tools', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libev-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common']) => {"ansible_loop_var": "item", "cache_update_time": 1571344640, "cache_updated": false, "changed": true, "item": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2019-10-17T20:42:19.042 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.4).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 88 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]\nFetched 10.4 MB in 4s (2690 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading 2019-10-17T20:42:19.044 INFO:teuthology.task.ansible.out:database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 136665 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest 2019-10-17T20:42:19.045 INFO:teuthology.task.ansible.out:version (1:2.17.1-1ubuntu0.4).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 88 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.19 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.19 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.19 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.19 [869 kB]", "Fetched 10.4 MB in 4s (2690 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 136665 files a 2019-10-17T20:42:19.047 INFO:teuthology.task.ansible.out:nd directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.19_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.19) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.19) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:19.048 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ non_aarch64_packages|list + non_aarch64_common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:19.094 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:19 +0000 (0:00:00.071) 0:02:05.532 ****** 2019-10-17T20:42:19.137 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via 2019-10-17T20:42:19.138 INFO:teuthology.task.ansible.out: squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ non_aarch64_packages|list + non_aarch64_common_packages|list }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:19.918 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-10-17T20:42:19.919 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=[u'libgoogle-perftools4', u'iozone3', u'libssl1.0.0', u'smbios-utils', u'libfcgi', u'sysprof']) => {"ansible_loop_var": "item", "cache_update_time": 1571344592, "cache_updated": false, "changed": false, "item": ["libgoogle-perftools4", "iozone3", "libssl1.0.0", "smbios-utils", "libfcgi", "sysprof"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:19.971 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:19 +0000 (0:00:00.876) 0:02:06.408 ****** 2019-10-17T20:42:20.097 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=[u'libgoogle-perftools4', u'iozone3', u'libssl1.0.0', u'smbios-utils', u'libfcgi', u'sysprof']) => {"ansible_loop_var": "item", "cache_update_time": 1571344640, "cache_updated": false, "changed": false, "item": ["libgoogle-perftools4", "iozone3", "libssl1.0.0", "smbios-utils", "libfcgi", "sysprof"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:20.147 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.176) 0:02:06.585 ****** 2019-10-17T20:42:20.154 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-17T20:42:20.156 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-10-17T20:42:20.207 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.059) 0:02:06.645 ****** 2019-10-17T20:42:20.230 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:20.235 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-17T20:42:20.236 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:20.288 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.081) 0:02:06.726 ****** 2019-10-17T20:42:20.294 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-10-17T20:42:20.295 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2019-10-17T20:42:20.339 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.051) 0:02:06.777 ****** 2019-10-17T20:42:20.362 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:20.366 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-10-17T20:42:20.368 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:20.408 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.068) 0:02:06.845 ****** 2019-10-17T20:42:20.424 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-10-17T20:42:20.425 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-10-17T20:42:20.476 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.068) 0:02:06.914 ****** 2019-10-17T20:42:20.556 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2019-10-17T20:42:20.597 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.120) 0:02:07.034 ****** 2019-10-17T20:42:20.806 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-10-17T20:42:20.808 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-10-17T20:42:20.857 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.259) 0:02:07.294 ****** 2019-10-17T20:42:20.959 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2019-10-17T20:42:21.003 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:20 +0000 (0:00:00.146) 0:02:07.440 ****** 2019-10-17T20:42:21.012 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-10-17T20:42:21.013 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-17T20:42:21.059 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.056) 0:02:07.497 ****** 2019-10-17T20:42:21.171 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1209, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-10-17T20:42:21.214 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.155) 0:02:07.652 ****** 2019-10-17T20:42:21.220 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-10-17T20:42:21.220 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:42:21.271 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.056) 0:02:07.709 ****** 2019-10-17T20:42:21.350 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:42:21.390 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.119) 0:02:07.828 ****** 2019-10-17T20:42:21.566 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-17T20:42:21.568 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2019-10-17 20:42:05 UTC", "ActiveEnterTimestampMonotonic": "138666833", "ActiveExitTimestamp": "Thu 2019-10-17 20:41:52 UTC", "ActiveExitTimestampMonotonic": "125425094", "ActiveState": "active", "After": "remote-fs.target -.mount systemd-journald.socket systemd-tmpfiles-setup.service network.target sysinit.target basic.target nss-lookup.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-10-17 20:42:05 UTC", "AssertTimestampMonotonic": "138641942", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-10-17 20:42:05 UTC", "ConditionTimestampMonotonic": "138641942", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", " 2019-10-17T20:42:21.570 INFO:teuthology.task.ansible.out:Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6633", "ExecMainStartTimestamp": "Thu 2019-10-17 20:42:05 UTC", "ExecMainStartTimestampMonotonic": "138666818", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Thu 2019-10-17 20:41:52 UTC", "InactiveEnterTimestampMonotonic": "125512417", "InactiveExitTimestamp": "Thu 2019-10-17 20:42:05 UTC", "InactiveExitTimestampMonotonic": "138642691", "InvocationID": "11443b43cd084418b66040a4610765bb", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "i 2019-10-17T20:42:21.571 INFO:teuthology.task.ansible.out:nfinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "6633", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDir 2019-10-17T20:42:21.573 INFO:teuthology.task.ansible.out:ectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-10-17 20:42:05 UTC", "StateChangeTimestampMonotonic": "138666833", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-10-17 20:42:05 UTC", "WatchdogTimestampMonotonic": "138666832", "WatchdogUSec": "0"}} 2019-10-17T20:42:21.616 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.226) 0:02:08.054 ****** 2019-10-17T20:42:21.645 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-17T20:42:21.646 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:21.696 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.079) 0:02:08.133 ****** 2019-10-17T20:42:21.704 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-10-17T20:42:21.708 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2019-10-17 20:42:06 UTC", "ActiveEnterTimestampMonotonic": "143349376", "ActiveExitTimestamp": "Thu 2019-10-17 20:41:53 UTC", "ActiveExitTimestampMonotonic": "131093404", "ActiveState": "active", "After": "-.mount sysinit.target network.target basic.target systemd-journald.socket remote-fs.target systemd-tmpfiles-setup.service system.slice nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-10-17 20:42:06 UTC", "AssertTimestampMonotonic": "143326454", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-10-17 20:42:06 UTC", "ConditionTimestampMonotonic": "143326454", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", " 2019-10-17T20:42:21.710 INFO:teuthology.task.ansible.out:Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "6632", "ExecMainStartTimestamp": "Thu 2019-10-17 20:42:06 UTC", "ExecMainStartTimestampMonotonic": "143349366", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Thu 2019-10-17 20:41:53 UTC", "InactiveEnterTimestampMonotonic": "131180192", "InactiveExitTimestamp": "Thu 2019-10-17 20:42:06 UTC", "InactiveExitTimestampMonotonic": "143327168", "InvocationID": "aa0a21af15de47b88cf2df85cceaee69", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "i 2019-10-17T20:42:21.711 INFO:teuthology.task.ansible.out:nfinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "6632", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDir 2019-10-17T20:42:21.713 INFO:teuthology.task.ansible.out:ectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-10-17 20:42:06 UTC", "StateChangeTimestampMonotonic": "143349376", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "Tasks 2019-10-17T20:42:21.715 INFO:teuthology.task.ansible.out:Accounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-10-17 20:42:06 UTC", "WatchdogTimestampMonotonic": "143349376", "WatchdogUSec": "0"}} 2019-10-17T20:42:21.716 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:21.759 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.063) 0:02:08.196 ****** 2019-10-17T20:42:21.798 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.039) 0:02:08.236 ****** 2019-10-17T20:42:21.805 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-17T20:42:21.806 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:21.850 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.051) 0:02:08.287 ****** 2019-10-17T20:42:21.855 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-10-17T20:42:21.855 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:21.898 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.048) 0:02:08.335 ****** 2019-10-17T20:42:21.903 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-17T20:42:21.904 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:21.952 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.053) 0:02:08.389 ****** 2019-10-17T20:42:21.958 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.003 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:21 +0000 (0:00:00.050) 0:02:08.440 ****** 2019-10-17T20:42:22.008 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-10-17T20:42:22.009 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.052 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.050) 0:02:08.490 ****** 2019-10-17T20:42:22.058 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-10-17T20:42:22.058 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.102 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.049) 0:02:08.540 ****** 2019-10-17T20:42:22.108 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-17T20:42:22.109 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.153 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.051) 0:02:08.591 ****** 2019-10-17T20:42:22.159 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-10-17T20:42:22.160 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.174 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.207 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.054) 0:02:08.645 ****** 2019-10-17T20:42:22.242 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi092.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.285 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.077) 0:02:08.722 ****** 2019-10-17T20:42:22.296 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.338 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.053) 0:02:08.776 ****** 2019-10-17T20:42:22.346 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-17T20:42:22.396 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.056) 0:02:08.833 ****** 2019-10-17T20:42:22.401 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-10-17T20:42:22.403 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.417 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.448 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.052) 0:02:08.885 ****** 2019-10-17T20:42:22.453 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-17T20:42:22.454 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.469 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ ceph_packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.501 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.053) 0:02:08.939 ****** 2019-10-17T20:42:22.517 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-10-17T20:42:22.519 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.521 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.573 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.072) 0:02:09.011 ****** 2019-10-17T20:42:22.579 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-17T20:42:22.581 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.628 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.054) 0:02:09.065 ****** 2019-10-17T20:42:22.635 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-10-17T20:42:22.650 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.682 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.054) 0:02:09.120 ****** 2019-10-17T20:42:22.689 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-17T20:42:22.691 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.705 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_remove }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.733 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.050) 0:02:09.171 ****** 2019-10-17T20:42:22.739 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-10-17T20:42:22.739 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.781 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.047) 0:02:09.218 ****** 2019-10-17T20:42:22.786 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-17T20:42:22.788 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.810 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.828 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.047) 0:02:09.265 ****** 2019-10-17T20:42:22.844 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-10-17T20:42:22.846 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.886 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.057) 0:02:09.322 ****** 2019-10-17T20:42:22.891 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-17T20:42:22.892 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.904 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "zypper" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ packages_to_upgrade }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:22.937 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.052) 0:02:09.375 ****** 2019-10-17T20:42:22.943 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-10-17T20:42:22.944 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=[]) => {"ansible_loop_var": "item", "changed": false, "item": [], "skip_reason": "Conditional result was False"} 2019-10-17T20:42:22.991 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:22 +0000 (0:00:00.053) 0:02:09.428 ****** 2019-10-17T20:42:22.996 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-17T20:42:22.998 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.038 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.475 ****** 2019-10-17T20:42:23.046 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.087 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.523 ****** 2019-10-17T20:42:23.092 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-17T20:42:23.092 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.135 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.048) 0:02:09.571 ****** 2019-10-17T20:42:23.142 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.183 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.619 ****** 2019-10-17T20:42:23.187 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-17T20:42:23.188 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.228 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.046) 0:02:09.666 ****** 2019-10-17T20:42:23.233 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-17T20:42:23.235 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.276 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.713 ****** 2019-10-17T20:42:23.282 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-17T20:42:23.284 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.323 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.761 ****** 2019-10-17T20:42:23.329 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-17T20:42:23.331 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.371 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.808 ****** 2019-10-17T20:42:23.377 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-17T20:42:23.377 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.418 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:09.856 ****** 2019-10-17T20:42:23.424 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-17T20:42:23.425 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.466 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.048) 0:02:09.904 ****** 2019-10-17T20:42:23.472 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-17T20:42:23.473 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.515 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.048) 0:02:09.952 ****** 2019-10-17T20:42:23.520 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-17T20:42:23.522 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.566 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.051) 0:02:10.004 ****** 2019-10-17T20:42:23.572 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-17T20:42:23.572 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.614 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.047) 0:02:10.052 ****** 2019-10-17T20:42:23.620 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-10-17T20:42:23.621 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.673 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.053) 0:02:10.105 ****** 2019-10-17T20:42:23.674 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-17T20:42:23.676 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.719 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.050) 0:02:10.156 ****** 2019-10-17T20:42:23.725 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-17T20:42:23.727 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.784 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.065) 0:02:10.222 ****** 2019-10-17T20:42:23.792 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-17T20:42:23.793 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.850 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.065) 0:02:10.288 ****** 2019-10-17T20:42:23.859 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.900 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.049) 0:02:10.337 ****** 2019-10-17T20:42:23.906 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-17T20:42:23.907 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:23.950 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:23 +0000 (0:00:00.050) 0:02:10.388 ****** 2019-10-17T20:42:23.958 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-17T20:42:23.959 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.006 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.055) 0:02:10.443 ****** 2019-10-17T20:42:24.011 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-17T20:42:24.012 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.056 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.050) 0:02:10.494 ****** 2019-10-17T20:42:24.064 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-10-17T20:42:24.065 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.116 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.059) 0:02:10.553 ****** 2019-10-17T20:42:24.122 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-17T20:42:24.123 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.171 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.054) 0:02:10.607 ****** 2019-10-17T20:42:24.176 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-10-17T20:42:24.177 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.249 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.078) 0:02:10.686 ****** 2019-10-17T20:42:24.276 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-10-17T20:42:24.278 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.329 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.080) 0:02:10.766 ****** 2019-10-17T20:42:24.334 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-17T20:42:24.339 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-10-17T20:42:24.380 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.051) 0:02:10.817 ****** 2019-10-17T20:42:24.387 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-10-17T20:42:24.389 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.431 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.050) 0:02:10.868 ****** 2019-10-17T20:42:24.587 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-10-17T20:42:24.589 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2019-10-17T20:42:24.629 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.198) 0:02:11.067 ****** 2019-10-17T20:42:24.723 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-17T20:42:24.724 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-10-17T20:42:24.766 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.136) 0:02:11.204 ****** 2019-10-17T20:42:24.794 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-17T20:42:24.796 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:24.836 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:24 +0000 (0:00:00.069) 0:02:11.273 ****** 2019-10-17T20:42:24.967 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-10-17T20:42:24.968 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2019-10-17T20:42:25.022 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.186) 0:02:11.460 ****** 2019-10-17T20:42:25.072 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-10-17T20:42:25.073 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:25.115 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.092) 0:02:11.553 ****** 2019-10-17T20:42:25.173 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-10-17T20:42:25.177 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-10-17T20:42:25.229 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.114) 0:02:11.667 ****** 2019-10-17T20:42:25.489 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2019-10-17T20:42:25.532 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.302) 0:02:11.969 ****** 2019-10-17T20:42:25.548 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-10-17T20:42:25.549 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-10-17T20:42:25.609 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.077) 0:02:12.047 ****** 2019-10-17T20:42:25.894 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2019-10-17T20:42:25.936 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.326) 0:02:12.373 ****** 2019-10-17T20:42:25.953 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-10-17T20:42:25.955 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-10-17T20:42:25.996 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:25 +0000 (0:00:00.059) 0:02:12.433 ****** 2019-10-17T20:42:26.142 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-17T20:42:26.149 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:42:26.184 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.188) 0:02:12.622 ****** 2019-10-17T20:42:26.210 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-10-17T20:42:26.212 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.252 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.065) 0:02:12.688 ****** 2019-10-17T20:42:26.276 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-17T20:42:26.278 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2019-10-17T20:42:26.318 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.067) 0:02:12.756 ****** 2019-10-17T20:42:26.358 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.040) 0:02:12.796 ****** 2019-10-17T20:42:26.364 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-10-17T20:42:26.366 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.407 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.048) 0:02:12.844 ****** 2019-10-17T20:42:26.433 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-10-17T20:42:26.434 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.474 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.067) 0:02:12.911 ****** 2019-10-17T20:42:26.501 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-10-17T20:42:26.503 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:42:26.549 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.075) 0:02:12.987 ****** 2019-10-17T20:42:26.555 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-17T20:42:26.556 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.598 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.048) 0:02:13.035 ****** 2019-10-17T20:42:26.605 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.646 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.048) 0:02:13.083 ****** 2019-10-17T20:42:26.651 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:42:26.652 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-17T20:42:26.694 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.047) 0:02:13.131 ****** 2019-10-17T20:42:26.699 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-10-17T20:42:26.700 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.751 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.057) 0:02:13.188 ****** 2019-10-17T20:42:26.758 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-17T20:42:26.759 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.761 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.762 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.774 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.775 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.831 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.080) 0:02:13.268 ****** 2019-10-17T20:42:26.837 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-17T20:42:26.840 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.843 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.844 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} ok: [smithi092.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-17T20:42:26.855 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.856 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.902 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.071) 0:02:13.339 ****** 2019-10-17T20:42:26.910 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi092.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.911 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:26.958 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:26 +0000 (0:00:00.055) 0:02:13.395 ****** 2019-10-17T20:42:26.964 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-10-17T20:42:26.966 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.012 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.054) 0:02:13.450 ****** 2019-10-17T20:42:27.061 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-10-17T20:42:27.102 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.090) 0:02:13.540 ****** 2019-10-17T20:42:27.119 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-17T20:42:27.120 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003810", "end": "2019-10-17 20:42:27.097894", "rc": 0, "start": "2019-10-17 20:42:27.094084", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-17T20:42:27.178 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.075) 0:02:13.616 ****** 2019-10-17T20:42:27.188 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-10-17T20:42:27.189 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.191 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.192 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.194 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.195 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.217 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.218 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi204.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.268 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.089) 0:02:13.705 ****** 2019-10-17T20:42:27.273 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-10-17T20:42:27.274 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.275 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.277 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.324 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.056) 0:02:13.761 ****** 2019-10-17T20:42:27.329 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-10-17T20:42:27.331 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.374 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.050) 0:02:13.812 ****** 2019-10-17T20:42:27.423 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-10-17T20:42:27.425 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.468 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.093) 0:02:13.906 ****** 2019-10-17T20:42:27.485 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-10-17T20:42:27.486 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003996", "end": "2019-10-17 20:42:27.464291", "rc": 0, "start": "2019-10-17 20:42:27.460295", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-10-17T20:42:27.487 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:27.530 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.061) 0:02:13.968 ****** 2019-10-17T20:42:27.559 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-10-17T20:42:27.560 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.605 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.074) 0:02:14.042 ****** 2019-10-17T20:42:27.642 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.682 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.076) 0:02:14.119 ****** 2019-10-17T20:42:27.711 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-10-17T20:42:27.712 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:27.759 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:27 +0000 (0:00:00.077) 0:02:14.196 ****** 2019-10-17T20:42:27.790 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "apt" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: ['gdisk', 'dmsetup']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:28.358 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-17T20:42:28.359 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=[u'gdisk', u'dmsetup']) => {"ansible_loop_var": "item", "cache_update_time": 1571344592, "cache_updated": false, "changed": false, "item": ["gdisk", "dmsetup"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:28.414 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.654) 0:02:14.851 ****** 2019-10-17T20:42:28.453 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-17T20:42:28.455 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:28.502 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.087) 0:02:14.939 ****** 2019-10-17T20:42:28.553 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi092.front.sepia.ceph.com] => (item={u'block_used': 13658419, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226436689, u'size_available': 927484678144, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906479, u'device': u'/dev/sda1', u'inode_used': 148497, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226436689, "block_size": 4096, "block_total": 240095108, "block_used": 13658419, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906479, "inode_total": 61054976, "inode_used": 148497, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927484678144, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}} 2019-10-17T20:42:28.607 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.105) 0:02:15.045 ****** 2019-10-17T20:42:28.698 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-10-17T20:42:28.699 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=[u'gdisk', u'dmsetup']) => {"ansible_loop_var": "item", "cache_update_time": 1571344640, "cache_updated": false, "changed": false, "item": ["gdisk", "dmsetup"], "warnings": ["Could not find aptitude. Using apt-get instead"]} 2019-10-17T20:42:28.743 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.135) 0:02:15.180 ****** 2019-10-17T20:42:28.763 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-17T20:42:28.764 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008254", "end": "2019-10-17 20:42:28.732348", "rc": 0, "start": "2019-10-17 20:42:28.724094", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:28.810 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.066) 0:02:15.247 ****** 2019-10-17T20:42:28.816 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-10-17T20:42:28.817 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=gdisk) => {"ansible_loop_var": "item", "changed": false, "item": "gdisk", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:28.818 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=device-mapper) => {"ansible_loop_var": "item", "changed": false, "item": "device-mapper", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:28.864 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.054) 0:02:15.302 ****** 2019-10-17T20:42:28.871 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-17T20:42:28.874 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'block_used': 13658419, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226436689, u'size_available': 927484678144, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906479, u'device': u'/dev/sda1', u'inode_used': 148497, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226436689, "block_size": 4096, "block_total": 240095108, "block_used": 13658419, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906479, "inode_total": 61054976, "inode_used": 148497, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927484678144, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:28.920 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.055) 0:02:15.358 ****** 2019-10-17T20:42:28.926 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-17T20:42:28.927 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item={u'block_used': 13654324, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440784, u'size_available': 927501451264, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440784, "block_size": 4096, "block_total": 240095108, "block_used": 13654324, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501451264, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}} 2019-10-17T20:42:28.969 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:28 +0000 (0:00:00.049) 0:02:15.407 ****** 2019-10-17T20:42:28.975 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-17T20:42:29.018 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:29 +0000 (0:00:00.048) 0:02:15.455 ****** 2019-10-17T20:42:29.071 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-17T20:42:29.074 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5LJ4S', u'wwn-0x5000c500918da5aa'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5LJ4S', u'wwn': u'0x5000c500918da5aa', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1', u'wwn-0x5000c500918da5aa-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S", "wwn-0x5000c500918da5aa"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1", "wwn-0x5000c500918da5aa-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJ4S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-17T20:42:29.075 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500918da5aa"}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.078 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.093 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.095 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.108 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.121 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.122 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-10-17T20:42:29.124 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009808", "end": "2019-10-17 20:42:29.247427", "rc": 0, "start": "2019-10-17 20:42:29.237619", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:29.175 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:29 +0000 (0:00:00.156) 0:02:15.612 ****** 2019-10-17T20:42:29.184 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-17T20:42:29.186 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.191 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.193 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.218 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-10-17T20:42:29.219 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'block_used': 13654324, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440784, u'size_available': 927501451264, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440784, "block_size": 4096, "block_total": 240095108, "block_used": 13654324, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501451264, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.262 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:29 +0000 (0:00:00.087) 0:02:15.700 ****** 2019-10-17T20:42:29.288 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-10-17T20:42:29.351 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:29 +0000 (0:00:00.088) 0:02:15.788 ****** 2019-10-17T20:42:29.402 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-10-17T20:42:29.403 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E', u'wwn-0x5000c50092941a36'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XP2E', u'wwn': u'0x5000c50092941a36', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1', u'wwn-0x5000c50092941a36-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-17T20:42:29.405 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50092941a36"}}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.416 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.428 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.440 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.453 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.465 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.477 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:29.489 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:31.060 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN', u'nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.763606", "end": "2019-10-17 20:42:31.030309", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN", "nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:29.266703", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-17T20:42:31.062 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:31.289 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN', u'nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.664959", "end": "2019-10-17 20:42:31.409118", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:29.744159", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2019-10-17T20:42:31.291 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:32.292 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.100023", "end": "2019-10-17 20:42:32.262460", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:31.162437", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:32.559 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.119353", "end": "2019-10-17 20:42:32.663513", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:31.544160", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:33.525 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.103874", "end": "2019-10-17 20:42:33.504434", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:32.400560", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:33.876 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.186037", "end": "2019-10-17 20:42:33.979844", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:32.793807", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:34.746 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.083820", "end": "2019-10-17 20:42:34.722518", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:33.638698", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:35.128 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.112197", "end": "2019-10-17 20:42:35.231104", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:34.118907", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:35.998 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X'], u'uuids': [u'a1e335b0-c538-47e5-8baf-4d144ca2ebf1']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.096818", "end": "2019-10-17 20:42:35.957932", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X"], "labels": [], "masters": [], "uuids": ["a1e335b0-c538-47e5-8baf-4d144ca2ebf1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:34.861114", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:36.334 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi'], u'uuids': [u'56e60058-f0cb-47e1-a3d6-bf071031bc28']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.088970", "end": "2019-10-17 20:42:36.445605", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi"], "labels": [], "masters": [], "uuids": ["56e60058-f0cb-47e1-a3d6-bf071031bc28"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:35.356635", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:37.208 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.080336", "end": "2019-10-17 20:42:37.168662", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:36.088326", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:37.261 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:37 +0000 (0:00:07.910) 0:02:23.699 ****** 2019-10-17T20:42:37.321 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-17T20:42:37.322 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5LJ4S', u'wwn-0x5000c500918da5aa'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5LJ4S', u'wwn': u'0x5000c500918da5aa', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1', u'wwn-0x5000c500918da5aa-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S", "wwn-0x5000c500918da5aa"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJ4S-part1", "wwn-0x5000c500918da5aa-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJ4S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-17T20:42:37.324 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500918da5aa"}}, "skip_reason": "Conditional result was False"} skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.335 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.347 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.358 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.371 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.374 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.388 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.398 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.570 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.102828", "end": "2019-10-17 20:42:37.672223", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:36.569395", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-10-17T20:42:37.626 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:37 +0000 (0:00:00.364) 0:02:24.063 ****** 2019-10-17T20:42:37.632 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-10-17T20:42:37.635 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN', u'nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.047062", "end": "2019-10-17 20:42:37.559320", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Yf04ul-n5Ut-2W9B-ssqE-QzLl-b4NK-jeBdfS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QM400BGN", "nvme-nvme.8086-50484654363230343030514d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:37.512258", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-10-17T20:42:37.637 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0403177 s, 2.9 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0403177 s, 2.9 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:37.680 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E', u'wwn-0x5000c50092941a36'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XP2E', u'wwn': u'0x5000c50092941a36', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1', u'wwn-0x5000c50092941a36-part1'], u'uuids': [u'00033e24-d094-4573-8712-cc5f9717bdc8']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E", "wwn-0x5000c50092941a36"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XP2E-part1", "wwn-0x5000c50092941a36-part1"], "labels": [], "masters": [], "uuids": ["00033e24-d094-4573-8712-cc5f9717bdc8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XP2E", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2019-10-17T20:42:37.683 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c50092941a36"}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.693 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.696 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.708 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.710 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:37.723 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.725 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.737 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.777 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:37.912 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.157413", "end": "2019-10-17 20:42:37.876920", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHPiuC1GN3qty3CR7b0bM3QBeqXRC8vplR"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:37.719507", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153999 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153999 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:37.928 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN', u'nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.048901", "end": "2019-10-17 20:42:38.038687", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7Ugq9W-TEnC-MMst-1bfZ-3kqk-PYMP-LdUkRS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GD400BGN", "nvme-nvme.8086-43564654363233333030474434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:37.989786", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2019-10-17T20:42:37.930 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0328999 s, 3.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0328999 s, 3.5 GB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:38.191 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.157442", "end": "2019-10-17 20:42:38.169600", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH4hOeEpx3NfY9Ug2kR5tNSMpg9jZtXUSZ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.012158", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152708 s, 755 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152708 s, 755 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:38.229 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.158304", "end": "2019-10-17 20:42:38.334946", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHM18VZXUQpi4qZ8JLaejeogsDLh8Vd6OnB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.176642", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153288 s, 752 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153288 s, 752 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:38.492 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.148839", "end": "2019-10-17 20:42:38.459687", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHQCqiF2cXTYWzK0MjeCFTvHoSOKKrwEJr"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.310848", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144591 s, 798 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144591 s, 798 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:38.506 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.145190", "end": "2019-10-17 20:42:38.616449", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMERZIak7JeuTBQjmWAtnATULUzBYn9QNN"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.471259", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141242 s, 817 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141242 s, 817 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:38.780 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X'], u'uuids': [u'a1e335b0-c538-47e5-8baf-4d144ca2ebf1']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.159195", "end": "2019-10-17 20:42:38.758284", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHHNmbmpcGdp6Q6P6iKkCjaTAXYPsl3YT7X"], "labels": [], "masters": [], "uuids": ["a1e335b0-c538-47e5-8baf-4d144ca2ebf1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.599089", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154525 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154525 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:38.792 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.150553", "end": "2019-10-17 20:42:38.907126", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMaAQoqwC4EjLl8ArjVKWIPr7i7oCSEddf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.756573", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146838 s, 786 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146838 s, 786 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.071 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156209", "end": "2019-10-17 20:42:39.038897", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LPWAqjCnrrc8XeEHaDsM0LiEClY2CKHH54qPdqH9VP1mY9PIpqVv9OyxFxHQEIGy"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:38.882688", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151712 s, 760 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151712 s, 760 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.096 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi'], u'uuids': [u'56e60058-f0cb-47e1-a3d6-bf071031bc28']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.148815", "end": "2019-10-17 20:42:39.209035", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMuucagXoAD2ME120lhMVAnWgQYodjekxi"], "labels": [], "masters": [], "uuids": ["56e60058-f0cb-47e1-a3d6-bf071031bc28"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:39.060220", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145125 s, 795 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145125 s, 795 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.141 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:01.514) 0:02:25.578 ****** 2019-10-17T20:42:39.376 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.148604", "end": "2019-10-17 20:42:39.483758", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-en9qxNnJCP5WDn6rXfqjO1UoXqviIcHMQMLsxKLk1ctE2dxaFHNrOxB3LNra6eqB"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-10-17 20:42:39.335154", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145578 s, 792 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145578 s, 792 MB/s"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.455 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:00.314) 0:02:25.893 ****** 2019-10-17T20:42:39.464 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi092.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.155150", "end": "2019-10-17 20:42:39.414395", "rc": 0, "start": "2019-10-17 20:42:39.259245", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.527 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:00.071) 0:02:25.964 ****** 2019-10-17T20:42:39.714 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-17T20:42:39.717 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051825", "end": "2019-10-17 20:42:39.693901", "rc": 0, "start": "2019-10-17 20:42:39.642076", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.770 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:00.243) 0:02:26.208 ****** 2019-10-17T20:42:39.780 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-10-17T20:42:39.782 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.140059", "end": "2019-10-17 20:42:39.848984", "rc": 0, "start": "2019-10-17 20:42:39.708925", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:39.847 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:00.077) 0:02:26.285 ****** 2019-10-17T20:42:39.855 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-17T20:42:39.913 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:00.065) 0:02:26.351 ****** 2019-10-17T20:42:39.942 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-17T20:42:39.944 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:39.986 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:39 +0000 (0:00:00.073) 0:02:26.424 ****** 2019-10-17T20:42:40.015 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-17T20:42:40.017 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.056 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.069) 0:02:26.494 ****** 2019-10-17T20:42:40.063 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-10-17T20:42:40.064 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.044804", "end": "2019-10-17 20:42:40.149468", "rc": 0, "start": "2019-10-17 20:42:40.104664", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:40.105 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.048) 0:02:26.542 ****** 2019-10-17T20:42:40.110 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-17T20:42:40.112 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.155 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.050) 0:02:26.593 ****** 2019-10-17T20:42:40.161 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-10-17T20:42:40.205 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.049) 0:02:26.642 ****** 2019-10-17T20:42:40.225 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.253 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.048) 0:02:26.691 ****** 2019-10-17T20:42:40.259 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-10-17T20:42:40.260 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.304 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.050) 0:02:26.741 ****** 2019-10-17T20:42:40.341 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-10-17T20:42:40.343 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.386 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.082) 0:02:26.824 ****** 2019-10-17T20:42:40.391 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:40.393 INFO:teuthology.task.ansible.out:TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-17T20:42:40.394 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:42:40.439 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.052) 0:02:26.877 ****** 2019-10-17T20:42:40.445 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-17T20:42:40.454 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.498 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.058) 0:02:26.935 ****** 2019-10-17T20:42:40.504 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-17T20:42:40.505 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={u'block_used': 13658419, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226436689, u'size_available': 927484678144, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906479, u'device': u'/dev/sda1', u'inode_used': 148497, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226436689, "block_size": 4096, "block_total": 240095108, "block_used": 13658419, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906479, "inode_total": 61054976, "inode_used": 148497, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927484678144, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.550 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.052) 0:02:26.988 ****** 2019-10-17T20:42:40.556 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-10-17T20:42:40.557 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.604 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.053) 0:02:27.041 ****** 2019-10-17T20:42:40.610 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-10-17T20:42:40.611 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.657 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.052) 0:02:27.094 ****** 2019-10-17T20:42:40.683 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:42:40.684 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.727 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.070) 0:02:27.165 ****** 2019-10-17T20:42:40.733 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-10-17T20:42:40.734 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-10-17T20:42:40.776 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.048) 0:02:27.213 ****** 2019-10-17T20:42:40.781 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:42:40.783 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.838 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.062) 0:02:27.276 ****** 2019-10-17T20:42:40.844 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-10-17T20:42:40.845 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={u'block_used': 13654324, u'uuid': u'00033e24-d094-4573-8712-cc5f9717bdc8', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226440784, u'size_available': 927501451264, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60906481, u'device': u'/dev/sda1', u'inode_used': 148495, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226440784, "block_size": 4096, "block_total": 240095108, "block_used": 13654324, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60906481, "inode_total": 61054976, "inode_used": 148495, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927501451264, "size_total": 983429562368, "uuid": "00033e24-d094-4573-8712-cc5f9717bdc8"}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.888 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.049) 0:02:27.326 ****** 2019-10-17T20:42:40.894 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-17T20:42:40.894 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.937 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.048) 0:02:27.375 ****** 2019-10-17T20:42:40.943 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-10-17T20:42:40.943 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:40.998 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:40 +0000 (0:00:00.055) 0:02:27.430 ****** 2019-10-17T20:42:41.000 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-10-17T20:42:41.002 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.043 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.050) 0:02:27.481 ****** 2019-10-17T20:42:41.052 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:42:41.054 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.095 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.051) 0:02:27.532 ****** 2019-10-17T20:42:41.101 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-17T20:42:41.102 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.158 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.058) 0:02:27.591 ****** 2019-10-17T20:42:41.164 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.202 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.048) 0:02:27.639 ****** 2019-10-17T20:42:41.228 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-10-17T20:42:41.230 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.271 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.068) 0:02:27.708 ****** 2019-10-17T20:42:41.307 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-10-17T20:42:41.309 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.350 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.079) 0:02:27.787 ****** 2019-10-17T20:42:41.377 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-10-17T20:42:41.378 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:41.416 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.066) 0:02:27.854 ****** 2019-10-17T20:42:41.732 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-10-17T20:42:41.734 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} [WARNING]: The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change. 2019-10-17T20:42:41.783 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.367) 0:02:28.221 ****** 2019-10-17T20:42:41.796 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}, "warnings": ["The value 4 (type int) in a string field was converted to u'4' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change."]} 2019-10-17T20:42:41.844 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:41 +0000 (0:00:00.060) 0:02:28.281 ****** 2019-10-17T20:42:42.346 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-10-17T20:42:42.348 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:42.360 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:42.601 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:42.602 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:42.817 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:42.838 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:43.057 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:43.101 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:43.263 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:43.317 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:43 +0000 (0:00:01.473) 0:02:29.755 ****** 2019-10-17T20:42:43.355 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-10-17T20:42:43.425 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:43 +0000 (0:00:00.108) 0:02:29.863 ****** 2019-10-17T20:42:43.463 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-10-17T20:42:43.465 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-17T20:42:43.562 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:43 +0000 (0:00:00.137) 0:02:30.000 ****** 2019-10-17T20:42:43.568 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-10-17T20:42:43.610 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:43 +0000 (0:00:00.047) 0:02:30.047 ****** 2019-10-17T20:42:43.616 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-10-17T20:42:43.617 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:43.670 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:43.714 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:43.784 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:43.849 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:43.903 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:43.988 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:44.043 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:44.117 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:44.172 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:44 +0000 (0:00:00.562) 0:02:30.610 ****** 2019-10-17T20:42:44.180 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-10-17T20:42:44.234 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:44 +0000 (0:00:00.061) 0:02:30.671 ****** 2019-10-17T20:42:44.313 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-10-17T20:42:44.315 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-10-17T20:42:44.361 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:44 +0000 (0:00:00.127) 0:02:30.799 ****** 2019-10-17T20:42:44.371 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2019-10-17T20:42:44.412 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:44 +0000 (0:00:00.050) 0:02:30.850 ****** 2019-10-17T20:42:44.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-10-17T20:42:44.418 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-10-17T20:42:44.464 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:44 +0000 (0:00:00.051) 0:02:30.901 ****** 2019-10-17T20:42:44.471 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-10-17T20:42:44.515 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:44 +0000 (0:00:00.050) 0:02:30.952 ****** 2019-10-17T20:42:44.975 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-10-17T20:42:44.977 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true} 2019-10-17T20:42:45.020 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:45 +0000 (0:00:00.505) 0:02:31.457 ****** 2019-10-17T20:42:45.622 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-17T20:42:45.624 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-17T20:42:45.694 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:45 +0000 (0:00:00.667) 0:02:32.125 ****** 2019-10-17T20:42:45.857 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-10-17T20:42:45.858 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-10-17T20:42:45.910 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:45 +0000 (0:00:00.222) 0:02:32.347 ****** 2019-10-17T20:42:45.989 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-10-17T20:42:45.991 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true} 2019-10-17T20:42:46.039 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.129) 0:02:32.477 ****** 2019-10-17T20:42:46.069 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-17T20:42:46.070 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-10-17T20:42:46.116 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.076) 0:02:32.553 ****** 2019-10-17T20:42:46.354 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-10-17T20:42:46.356 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-10-17T20:42:46.402 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.286) 0:02:32.839 ****** 2019-10-17T20:42:46.482 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-10-17T20:42:46.483 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-10-17T20:42:46.533 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.130) 0:02:32.970 ****** 2019-10-17T20:42:46.564 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2019-10-17T20:42:46.604 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.070) 0:02:33.041 ****** 2019-10-17T20:42:46.789 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-10-17T20:42:46.790 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2019-10-17T20:42:46.830 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.225) 0:02:33.267 ****** 2019-10-17T20:42:46.888 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-17T20:42:46.890 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-10-17T20:42:46.939 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:46 +0000 (0:00:00.109) 0:02:33.377 ****** 2019-10-17T20:42:47.178 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-10-17T20:42:47.179 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1209, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2019-10-17T20:42:47.231 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:47 +0000 (0:00:00.291) 0:02:33.668 ****** 2019-10-17T20:42:47.594 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-10-17T20:42:47.594 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2019-10-17T20:42:47.670 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:47 +0000 (0:00:00.439) 0:02:34.108 ****** 2019-10-17T20:42:47.822 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-17T20:42:47.823 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:42:47.865 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:47 +0000 (0:00:00.195) 0:02:34.303 ****** 2019-10-17T20:42:48.051 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-17T20:42:48.052 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.037994", "end": "2019-10-17 20:42:48.023516", "rc": 0, "start": "2019-10-17 20:42:47.985522", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-10-17T20:42:48.099 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:48 +0000 (0:00:00.233) 0:02:34.536 ****** 2019-10-17T20:42:48.137 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-17T20:42:48.138 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:48.184 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:48 +0000 (0:00:00.085) 0:02:34.622 ****** 2019-10-17T20:42:48.211 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-17T20:42:48.212 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:48.253 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:48 +0000 (0:00:00.068) 0:02:34.691 ****** 2019-10-17T20:42:48.565 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-10-17T20:42:48.566 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:42:48.608 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:48 +0000 (0:00:00.355) 0:02:35.046 ****** 2019-10-17T20:42:48.793 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-10-17T20:42:48.794 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.043589", "end": "2019-10-17 20:42:48.905456", "rc": 0, "start": "2019-10-17 20:42:48.861867", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2019-10-17T20:42:48.854 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:48 +0000 (0:00:00.245) 0:02:35.292 ****** 2019-10-17T20:42:48.892 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-10-17T20:42:48.893 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:48.936 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:48 +0000 (0:00:00.081) 0:02:35.374 ****** 2019-10-17T20:42:48.965 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-10-17T20:42:48.969 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:49.023 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:49 +0000 (0:00:00.086) 0:02:35.460 ****** 2019-10-17T20:42:49.093 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-10-17T20:42:49.094 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:42:49.139 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:49 +0000 (0:00:00.116) 0:02:35.577 ****** 2019-10-17T20:42:49.172 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:49.218 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:49 +0000 (0:00:00.078) 0:02:35.655 ****** 2019-10-17T20:42:49.574 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-10-17T20:42:49.625 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:49 +0000 (0:00:00.406) 0:02:36.062 ****** 2019-10-17T20:42:49.625 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:49.675 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:49.676 INFO:teuthology.task.ansible.out:TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:49.720 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:49 +0000 (0:00:00.094) 0:02:36.157 ****** 2019-10-17T20:42:49.722 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:49.931 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-10-17 20:40:13 UTC", "ActiveEnterTimestampMonotonic": "26858461", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket network.target basic.target system.slice sysinit.target -.mount systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-10-17 20:40:06 UTC", "AssertTimestampMonotonic": "18488071", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-10-17 20:40:06 UTC", "ConditionTimestampMonotonic": "18488070", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Dele 2019-10-17T20:42:49.933 INFO:teuthology.task.ansible.out:gate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1214", "ExecMainStartTimestamp": "Thu 2019-10-17 20:40:13 UTC", "ExecMainStartTimestampMonotonic": "26858435", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-10-17 20:40:06 UTC", "InactiveExitTimestampMonotonic": "18488662", "InvocationID": "e3a64a0f50484166acc1c6a9a580dc61", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infi 2019-10-17T20:42:49.934 INFO:teuthology.task.ansible.out:nity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1214", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", 2019-10-17T20:42:49.936 INFO:teuthology.task.ansible.out: "StateChangeTimestamp": "Thu 2019-10-17 20:40:13 UTC", "StateChangeTimestampMonotonic": "26858461", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-10-17 20:40:13 UTC", "WatchdogTimestampMonotonic": "26858460", "WatchdogUSec": "0"}} 2019-10-17T20:42:49.980 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:49 +0000 (0:00:00.259) 0:02:36.417 ****** 2019-10-17T20:42:49.981 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:49.989 INFO:teuthology.task.ansible.out: 2019-10-17T20:42:49.990 INFO:teuthology.task.ansible.out:TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:42:50.053 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.073) 0:02:36.491 ****** 2019-10-17T20:42:50.062 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-17T20:42:50.063 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:50.114 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.060) 0:02:36.552 ****** 2019-10-17T20:42:50.122 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:50.166 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.051) 0:02:36.603 ****** 2019-10-17T20:42:50.504 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-10-17T20:42:50.505 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2019-10-17T20:42:50.549 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.382) 0:02:36.986 ****** 2019-10-17T20:42:50.605 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-10-17T20:42:50.607 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:50.645 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.096) 0:02:37.083 ****** 2019-10-17T20:42:50.856 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-10-17T20:42:50.858 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-10-17 20:40:07 UTC", "ActiveEnterTimestampMonotonic": "24948748", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service basic.target network.target systemd-journald.socket system.slice sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-10-17 20:39:59 UTC", "AssertTimestampMonotonic": "16680635", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-10-17 20:39:59 UTC", "ConditionTimestampMonotonic": "16680635", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2019-10-17T20:42:50.860 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1160", "ExecMainStartTimestamp": "Thu 2019-10-17 20:40:07 UTC", "ExecMainStartTimestampMonotonic": "24948720", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-10-17 20:39:59 UTC", "InactiveExitTimestampMonotonic": "16681364", "InvocationID": "731cb2c7c4404949a3dc8986da38f334", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128034", "LimitNPROCSoft": "128034", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2019-10-17T20:42:50.861 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128034", "LimitSIGPENDINGSoft": "128034", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1160", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2019-10-17 20:40:07 UTC", "StateChangeTimestampMonoton 2019-10-17T20:42:50.863 INFO:teuthology.task.ansible.out:ic": "24948748", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2019-10-17 20:40:07 UTC", "WatchdogTimestampMonotonic": "24948747", "WatchdogUSec": "0"}} 2019-10-17T20:42:50.903 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.257) 0:02:37.340 ****** 2019-10-17T20:42:50.940 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-17T20:42:50.940 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"cache_update_time": 1571344592, "cache_updated": false, "changed": false} 2019-10-17T20:42:50.981 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:50 +0000 (0:00:00.077) 0:02:37.418 ****** 2019-10-17T20:42:50.987 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-10-17T20:42:50.989 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:51.032 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:51 +0000 (0:00:00.051) 0:02:37.470 ****** 2019-10-17T20:42:51.111 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-17T20:42:51.112 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-10-17T20:42:51.154 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:51 +0000 (0:00:00.121) 0:02:37.592 ****** 2019-10-17T20:42:51.620 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-10-17T20:42:51.620 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-10-17T20:42:51.661 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:51 +0000 (0:00:00.507) 0:02:38.099 ****** 2019-10-17T20:42:51.683 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:51.887 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-10-17T20:42:51.888 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"cache_update_time": 1571344640, "cache_updated": false, "changed": false} 2019-10-17T20:42:51.929 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:51 +0000 (0:00:00.267) 0:02:38.367 ****** 2019-10-17T20:42:52.071 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-10-17T20:42:52.071 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2019-10-17T20:42:52.122 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:52 +0000 (0:00:00.192) 0:02:38.559 ****** 2019-10-17T20:42:52.533 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-10-17T20:42:52.534 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1209, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2019-10-17T20:42:52.575 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:52 +0000 (0:00:00.453) 0:02:39.012 ****** 2019-10-17T20:42:52.598 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying `name: "{{ item }}"`, please use `name: '{{ pip_packages_to_install }}'` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:52.686 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-10-17T20:42:52.687 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-17T20:42:52.729 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:52 +0000 (0:00:00.154) 0:02:39.167 ****** 2019-10-17T20:42:53.213 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-10-17T20:42:53.214 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false} 2019-10-17T20:42:53.266 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:53 +0000 (0:00:00.536) 0:02:39.704 ****** 2019-10-17T20:42:53.307 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-17T20:42:53.310 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-17T20:42:53.354 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:53 +0000 (0:00:00.087) 0:02:39.792 ****** 2019-10-17T20:42:53.370 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-10-17T20:42:53.371 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=[u'remoto>=0.0.35']) => {"ansible_loop_var": "item", "changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "item": ["remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-10-17T20:42:53.415 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:53 +0000 (0:00:00.060) 0:02:39.853 ****** 2019-10-17T20:42:53.506 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-17T20:42:53.507 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1571344572.5597029, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1571344571.0884387, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 112104, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1571344571.0884387, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1832623006", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-10-17T20:42:53.566 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:53 +0000 (0:00:00.150) 0:02:40.004 ****** 2019-10-17T20:42:53.952 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi092.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-10-17T20:42:54.019 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.452) 0:02:40.456 ****** 2019-10-17T20:42:54.036 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-10-17T20:42:54.037 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false} 2019-10-17T20:42:54.045 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:54.095 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.075) 0:02:40.532 ****** 2019-10-17T20:42:54.101 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-17T20:42:54.102 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-17T20:42:54.152 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.057) 0:02:40.589 ****** 2019-10-17T20:42:54.159 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-10-17T20:42:54.159 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-10-17T20:42:54.202 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.050) 0:02:40.639 ****** 2019-10-17T20:42:54.207 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-10-17T20:42:54.208 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.251 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.048) 0:02:40.688 ****** 2019-10-17T20:42:54.344 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-10-17T20:42:54.345 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1571344553.1523938, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1571344551.3732839, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 112104, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1571344551.3732839, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1832623006", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2019-10-17T20:42:54.387 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.135) 0:02:40.823 ****** 2019-10-17T20:42:54.392 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-10-17T20:42:54.392 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-17T20:42:54.440 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.053) 0:02:40.877 ****** 2019-10-17T20:42:54.530 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-10-17T20:42:54.532 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2019-10-17T20:42:54.573 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.132) 0:02:41.010 ****** 2019-10-17T20:42:54.589 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-10-17T20:42:54.590 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-17T20:42:54.591 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|changed` use `result is changed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2019-10-17T20:42:54.634 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.061) 0:02:41.072 ****** 2019-10-17T20:42:54.640 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-10-17T20:42:54.643 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-10-17T20:42:54.683 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.048) 0:02:41.121 ****** 2019-10-17T20:42:54.688 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-17T20:42:54.689 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.730 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.046) 0:02:41.168 ****** 2019-10-17T20:42:54.736 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-10-17T20:42:54.738 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.786 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.053) 0:02:41.222 ****** 2019-10-17T20:42:54.790 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-17T20:42:54.790 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.792 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.793 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.804 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.805 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.848 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.064) 0:02:41.286 ****** 2019-10-17T20:42:54.886 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-17T20:42:54.886 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.888 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.914 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.916 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.917 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:54.920 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-10-17T20:42:54.964 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:54 +0000 (0:00:00.115) 0:02:41.402 ****** 2019-10-17T20:42:55.005 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.041) 0:02:41.443 ****** 2019-10-17T20:42:55.011 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:42:55.013 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.057 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.051) 0:02:41.494 ****** 2019-10-17T20:42:55.085 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-10-17T20:42:55.087 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.137 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.080) 0:02:41.575 ****** 2019-10-17T20:42:55.155 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-10-17T20:42:55.156 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-10-17T20:42:55.197 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.060) 0:02:41.635 ****** 2019-10-17T20:42:55.202 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-17T20:42:55.204 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.245 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.047) 0:02:41.683 ****** 2019-10-17T20:42:55.251 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-10-17T20:42:55.252 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.292 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.046) 0:02:41.729 ****** 2019-10-17T20:42:55.298 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-17T20:42:55.300 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.352 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.060) 0:02:41.789 ****** 2019-10-17T20:42:55.358 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-10-17T20:42:55.358 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.360 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.361 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.402 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.050) 0:02:41.840 ****** 2019-10-17T20:42:55.407 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-17T20:42:55.408 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.455 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.052) 0:02:41.893 ****** 2019-10-17T20:42:55.462 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-10-17T20:42:55.463 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.465 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.466 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.468 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.508 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.053) 0:02:41.946 ****** 2019-10-17T20:42:55.514 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-17T20:42:55.515 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.558 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.049) 0:02:41.996 ****** 2019-10-17T20:42:55.563 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-10-17T20:42:55.565 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.615 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.056) 0:02:42.052 ****** 2019-10-17T20:42:55.621 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-10-17T20:42:55.622 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.667 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.051) 0:02:42.104 ****** 2019-10-17T20:42:55.673 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-10-17T20:42:55.674 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.715 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.048) 0:02:42.152 ****** 2019-10-17T20:42:55.720 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-17T20:42:55.721 INFO:teuthology.task.ansible.out:skipping: [smithi092.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.763 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.048) 0:02:42.201 ****** 2019-10-17T20:42:55.769 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-10-17T20:42:55.771 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.815 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.051) 0:02:42.253 ****** 2019-10-17T20:42:55.851 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-10-17T20:42:55.853 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.896 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.081) 0:02:42.334 ****** 2019-10-17T20:42:55.923 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-10-17T20:42:55.925 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:55.974 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:55 +0000 (0:00:00.077) 0:02:42.411 ****** 2019-10-17T20:42:55.990 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi092.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010828", "end": "2019-10-17 20:42:55.905425", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-17 20:42:55.894597", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.007 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-17T20:42:56.008 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:56.056 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.081) 0:02:42.493 ****** 2019-10-17T20:42:56.087 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-10-17T20:42:56.089 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi092.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005983", "end": "2019-10-17 20:42:56.055489", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-17 20:42:56.049506", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.165 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.109) 0:02:42.602 ****** 2019-10-17T20:42:56.193 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-10-17T20:42:56.196 INFO:teuthology.task.ansible.out:skipping: [smithi204.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-10-17T20:42:56.247 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.081) 0:02:42.684 ****** 2019-10-17T20:42:56.253 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-17T20:42:56.254 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005493", "end": "2019-10-17 20:42:56.194258", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-17 20:42:56.188765", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.370 INFO:teuthology.task.ansible.out:ok: [smithi092.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015905", "end": "2019-10-17 20:42:56.333003", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-17 20:42:56.317098", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.415 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.167) 0:02:42.852 ****** 2019-10-17T20:42:56.433 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009131", "end": "2019-10-17 20:42:56.496361", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-10-17 20:42:56.487230", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.445 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-17T20:42:56.486 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.071) 0:02:42.923 ****** 2019-10-17T20:42:56.554 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-17T20:42:56.556 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004245", "end": "2019-10-17 20:42:56.653884", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-10-17 20:42:56.649639", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.619 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-17T20:42:56.621 INFO:teuthology.task.ansible.out:changed: [smithi092.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-10-17T20:42:56.700 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-10-17T20:42:56.701 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005342", "end": "2019-10-17 20:42:56.784745", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-10-17 20:42:56.779403", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.826 INFO:teuthology.task.ansible.out:ok: [smithi204.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.008959", "end": "2019-10-17 20:42:56.922908", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-10-17 20:42:56.913949", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-10-17T20:42:56.867 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.381) 0:02:43.304 ****** 2019-10-17T20:42:56.892 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-10-17T20:42:56.932 INFO:teuthology.task.ansible.out:Thursday 17 October 2019 20:42:56 +0000 (0:00:00.064) 0:02:43.369 ****** 2019-10-17T20:42:57.070 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-10-17T20:42:57.071 INFO:teuthology.task.ansible.out:changed: [smithi204.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2019-10-17T20:42:57.172 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-10-17T20:42:57.174 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2019-10-17T20:42:57.176 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2019-10-17T20:42:57.178 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi092.front.sepia.ceph.com : ok=111 changed=30 unreachable=0 failed=0 skipped=165 rescued=0 ignored=0 smithi204.front.sepia.ceph.com : ok=111 changed=30 unreachable=0 failed=0 skipped=165 rescued=0 ignored=0 Thursday 17 October 2019 20:42:57 +0000 (0:00:00.245) 0:02:43.614 ****** =============================================================================== 2019-10-17T20:42:57.184 INFO:teuthology.task.ansible.out:testnode : Upgrade packages -------------------------------------------- 22.20s users : Create all admin users with sudo access. ----------------------- 17.80s users : Update authorized_keys using the keys repo --------------------- 13.92s 2019-10-17T20:42:57.186 INFO:teuthology.task.ansible.out:testnode : Install packages -------------------------------------------- 12.02s testnode : Ensure packages are not present. ----------------------------- 8.72s testnode : Zap all non-root disks --------------------------------------- 7.91s Check for /usr/bin/python ----------------------------------------------- 1.97s users : Update authorized_keys for each user with literal keys ---------- 1.59s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.51s testnode : Create logical volume(s) ------------------------------------- 1.47s ansible-managed : Add authorized keys for the ansible user. ------------- 1.43s users : Remove revoked users -------------------------------------------- 1.30s Gathering Facts --------------------------------------------------------- 1.26s testnode : Install python-apt ------------------------------------------- 0.93s common : Update apt cache ----------------------------------------------- 0.89s testnode : Install non aarch64 packages --------------------------------- 0.88s testnode : Update apt cache. -------------------------------------------- 0.86s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.83s Gathering Facts --------------------------------------------------------- 0.80s common : Install Kerberos Packages (Debian) ----------------------------- 0.73s 2019-10-17T20:42:57.464 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2019-10-17T20:42:57.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:42:57.467 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi092.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:42:57.562 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:42:57.562 INFO:teuthology.orchestra.run.smithi092:> true 2019-10-17T20:42:57.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-10-17T20:42:57.740 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60} 2019-10-17T20:42:57.824 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:42:57.825 INFO:teuthology.orchestra.run.smithi204:> true 2019-10-17T20:42:58.030 INFO:teuthology.run_tasks:Running task clock... 2019-10-17T20:42:58.046 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-10-17T20:42:58.047 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:42:58.047 INFO:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-17T20:42:58.112 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-10-17T20:42:58.112 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Command line: ntpd -gq 2019-10-17T20:42:58.113 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: proto: precision = 0.124 usec (-23) 2019-10-17T20:42:58.114 INFO:teuthology.orchestra.run.smithi092.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stderr:restrict ::: KOD does nothing without LIMITED. 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: restrict ::: KOD does nothing without LIMITED. 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listen and drop on 0 v6wildcard [::]:123 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listen normally on 2 lo 127.0.0.1:123 2019-10-17T20:42:58.115 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listen normally on 3 enp3s0f1 172.21.15.92:123 2019-10-17T20:42:58.116 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listen normally on 4 lo [::1]:123 2019-10-17T20:42:58.116 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cd45%5]:123 2019-10-17T20:42:58.116 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:42:58 ntpd[7847]: Listening on routing socket on fd #22 for interface updates 2019-10-17T20:42:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:42:59 ntpd[7847]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:42:59.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:42:59 ntpd[7847]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:42:59.114 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:42:59 ntpd[7847]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:42:59.114 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:42:59 ntpd[7847]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:01.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:01 ntpd[7847]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:01.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:01 ntpd[7847]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:01.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:01 ntpd[7847]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:01.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:01 ntpd[7847]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:03.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:03 ntpd[7847]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:03.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:03 ntpd[7847]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:03.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:03 ntpd[7847]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:03.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:03 ntpd[7847]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:05.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:05 ntpd[7847]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:05.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:05 ntpd[7847]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:05.113 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:05 ntpd[7847]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:05.114 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:05 ntpd[7847]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:05.114 INFO:teuthology.orchestra.run.smithi092.stderr:17 Oct 20:43:05 ntpd[7847]: can't open /var/log/ntpstats/loopstats.20191017: Permission denied 2019-10-17T20:43:05.114 INFO:teuthology.orchestra.run.smithi092.stdout:17 Oct 20:43:05 ntpd[7847]: ntpd: time slew -0.005037 s 2019-10-17T20:43:05.114 INFO:teuthology.orchestra.run.smithi092.stdout:ntpd: time slew -0.005037s 2019-10-17T20:43:05.175 INFO:teuthology.orchestra.run.smithi092.stdout: remote refid st t when poll reach delay offset jitter 2019-10-17T20:43:05.175 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================== 2019-10-17T20:43:05.175 INFO:teuthology.orchestra.run.smithi092.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:05.176 INFO:teuthology.orchestra.run.smithi092.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:05.176 INFO:teuthology.orchestra.run.smithi092.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:05.176 INFO:teuthology.orchestra.run.smithi092.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:05.177 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:05.177 INFO:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-10-17T20:43:05.225 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: ntpd 4.2.8p10@1.3728-o (1): Starting 2019-10-17T20:43:05.225 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Command line: ntpd -gq 2019-10-17T20:43:05.230 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: proto: precision = 0.120 usec (-23) 2019-10-17T20:43:05.230 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-17T20:43:05.230 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: restrict ::: KOD does nothing without LIMITED. 2019-10-17T20:43:05.230 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listen and drop on 0 v6wildcard [::]:123 2019-10-17T20:43:05.230 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2019-10-17T20:43:05.231 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listen normally on 2 lo 127.0.0.1:123 2019-10-17T20:43:05.231 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listen normally on 3 enp3s0f1 172.21.15.204:123 2019-10-17T20:43:05.231 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listen normally on 4 lo [::1]:123 2019-10-17T20:43:05.231 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ab%5]:123 2019-10-17T20:43:05.231 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:05 ntpd[7850]: Listening on routing socket on fd #22 for interface updates 2019-10-17T20:43:05.231 INFO:teuthology.orchestra.run.smithi204.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2019-10-17T20:43:05.232 INFO:teuthology.orchestra.run.smithi204.stderr:restrict ::: KOD does nothing without LIMITED. 2019-10-17T20:43:06.226 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:06 ntpd[7850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:06.227 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:06 ntpd[7850]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:06.227 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:06 ntpd[7850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:06.227 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:06 ntpd[7850]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:08.227 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:08 ntpd[7850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:08.227 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:08 ntpd[7850]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:08.228 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:08 ntpd[7850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:08.228 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:08 ntpd[7850]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:10.228 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:10 ntpd[7850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:10.228 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:10 ntpd[7850]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:10.229 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:10 ntpd[7850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:10.229 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:10 ntpd[7850]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:12.229 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:12 ntpd[7850]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2019-10-17T20:43:12.230 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:12 ntpd[7850]: can't open /var/log/ntpstats/rawstats.20191017: Permission denied 2019-10-17T20:43:12.230 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:12 ntpd[7850]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2019-10-17T20:43:12.230 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:12 ntpd[7850]: can't open /var/log/ntpstats/peerstats.20191017: Permission denied 2019-10-17T20:43:12.230 INFO:teuthology.orchestra.run.smithi204.stderr:17 Oct 20:43:12 ntpd[7850]: can't open /var/log/ntpstats/loopstats.20191017: Permission denied 2019-10-17T20:43:12.230 INFO:teuthology.orchestra.run.smithi204.stdout:17 Oct 20:43:12 ntpd[7850]: ntpd: time set -0.121923 s 2019-10-17T20:43:12.231 INFO:teuthology.orchestra.run.smithi204.stdout:ntpd: time set -0.121923s 2019-10-17T20:43:12.320 INFO:teuthology.orchestra.run.smithi204.stdout: remote refid st t when poll reach delay offset jitter 2019-10-17T20:43:12.320 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================== 2019-10-17T20:43:12.320 INFO:teuthology.orchestra.run.smithi204.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:12.321 INFO:teuthology.orchestra.run.smithi204.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:12.321 INFO:teuthology.orchestra.run.smithi204.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:12.321 INFO:teuthology.orchestra.run.smithi204.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-10-17T20:43:12.322 INFO:teuthology.run_tasks:Running task install... 2019-10-17T20:43:12.334 DEBUG:teuthology.task.install:project ceph 2019-10-17T20:43:12.335 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5'}} 2019-10-17T20:43:12.335 DEBUG:teuthology.task.install:config {'sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5'} 2019-10-17T20:43:12.335 INFO:teuthology.task.install:Using flavor: basic 2019-10-17T20:43:12.340 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-10-17T20:43:12.340 INFO:teuthology.task.install:extra packages: [] 2019-10-17T20:43:12.340 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:43:12.340 INFO:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2019-10-17T20:43:12.392 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2019-10-17T20:43:12.397 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-17T20:43:12.397 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:12.398 INFO:teuthology.orchestra.run.smithi204:> sudo lsb_release -is 2019-10-17T20:43:12.398 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:43:12.398 INFO:teuthology.orchestra.run.smithi092:> sudo apt-key list | grep Ceph 2019-10-17T20:43:12.453 INFO:teuthology.orchestra.run.smithi204.stdout:Ubuntu 2019-10-17T20:43:12.459 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-17T20:43:12.459 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:12.459 INFO:teuthology.orchestra.run.smithi204:> sudo apt-key list | grep Ceph 2019-10-17T20:43:12.488 INFO:teuthology.orchestra.run.smithi092.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-10-17T20:43:12.533 INFO:teuthology.orchestra.run.smithi092.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-10-17T20:43:12.533 INFO:teuthology.orchestra.run.smithi092.stdout:uid [ unknown] Ceph.com (release key) 2019-10-17T20:43:12.535 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:43:12.535 INFO:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2019-10-17T20:43:12.540 INFO:teuthology.orchestra.run.smithi204.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2019-10-17T20:43:12.584 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2019-10-17T20:43:12.588 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2019-10-17T20:43:12.589 INFO:teuthology.orchestra.run.smithi204.stdout:uid [ unknown] Ceph.com (release key) 2019-10-17T20:43:12.590 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:12.590 INFO:teuthology.orchestra.run.smithi204:> sudo lsb_release -is 2019-10-17T20:43:12.591 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-17T20:43:12.591 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2019-10-17T20:43:12.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 2019-10-17T20:43:12.648 INFO:teuthology.orchestra.run.smithi204.stdout:Ubuntu 2019-10-17T20:43:12.653 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-17T20:43:12.653 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-daemon, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2019-10-17T20:43:12.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 2019-10-17T20:43:12.761 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default/ 2019-10-17T20:43:12.762 INFO:teuthology.task.install.deb:Package version is 15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:43:12.807 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default/ 2019-10-17T20:43:12.808 INFO:teuthology.task.install.deb:Package version is 15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:43:12.881 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:43:12.881 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-10-17T20:43:12.923 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:12.924 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2019-10-17T20:43:12.931 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:43:12.931 INFO:teuthology.orchestra.run.smithi092:> sudo apt-get update 2019-10-17T20:43:12.974 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:12.974 INFO:teuthology.orchestra.run.smithi204:> sudo apt-get update 2019-10-17T20:43:13.231 INFO:teuthology.orchestra.run.smithi092.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic InRelease 2019-10-17T20:43:13.242 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic InRelease 2019-10-17T20:43:13.245 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-17T20:43:13.257 INFO:teuthology.orchestra.run.smithi092.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-17T20:43:13.257 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-10-17T20:43:13.263 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2019-10-17T20:43:13.278 INFO:teuthology.orchestra.run.smithi092.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic Release.gpg 2019-10-17T20:43:13.285 INFO:teuthology.orchestra.run.smithi204.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic Release.gpg 2019-10-17T20:43:13.299 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.3 kB] 2019-10-17T20:43:13.303 INFO:teuthology.orchestra.run.smithi204.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2019-10-17T20:43:13.303 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2019-10-17T20:43:13.306 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.3 kB] 2019-10-17T20:43:13.329 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,875 B] 2019-10-17T20:43:13.340 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,875 B] 2019-10-17T20:43:13.406 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-17T20:43:13.452 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2019-10-17T20:43:13.784 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-17T20:43:13.833 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2019-10-17T20:43:14.179 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 279 kB in 1s (275 kB/s) 2019-10-17T20:43:14.242 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 279 kB in 1s (268 kB/s) 2019-10-17T20:43:15.302 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:43:15.346 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:43:15.347 INFO:teuthology.orchestra.run.smithi092:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-6144-gf0009d4-1bionic ceph-daemon=15.0.0-6144-gf0009d4-1bionic ceph-mds=15.0.0-6144-gf0009d4-1bionic ceph-mgr=15.0.0-6144-gf0009d4-1bionic ceph-common=15.0.0-6144-gf0009d4-1bionic ceph-fuse=15.0.0-6144-gf0009d4-1bionic ceph-test=15.0.0-6144-gf0009d4-1bionic radosgw=15.0.0-6144-gf0009d4-1bionic python-ceph=15.0.0-6144-gf0009d4-1bionic libcephfs2=15.0.0-6144-gf0009d4-1bionic libcephfs-dev=15.0.0-6144-gf0009d4-1bionic librados2=15.0.0-6144-gf0009d4-1bionic librbd1=15.0.0-6144-gf0009d4-1bionic rbd-fuse=15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:43:15.359 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:43:15.404 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:43:15.405 INFO:teuthology.orchestra.run.smithi204:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-6144-gf0009d4-1bionic ceph-daemon=15.0.0-6144-gf0009d4-1bionic ceph-mds=15.0.0-6144-gf0009d4-1bionic ceph-mgr=15.0.0-6144-gf0009d4-1bionic ceph-common=15.0.0-6144-gf0009d4-1bionic ceph-fuse=15.0.0-6144-gf0009d4-1bionic ceph-test=15.0.0-6144-gf0009d4-1bionic radosgw=15.0.0-6144-gf0009d4-1bionic python-ceph=15.0.0-6144-gf0009d4-1bionic libcephfs2=15.0.0-6144-gf0009d4-1bionic libcephfs-dev=15.0.0-6144-gf0009d4-1bionic librados2=15.0.0-6144-gf0009d4-1bionic librbd1=15.0.0-6144-gf0009d4-1bionic rbd-fuse=15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:43:15.423 INFO:teuthology.orchestra.run.smithi092.stdout:Reading package lists... 2019-10-17T20:43:15.487 INFO:teuthology.orchestra.run.smithi204.stdout:Reading package lists... 2019-10-17T20:43:15.671 INFO:teuthology.orchestra.run.smithi092.stdout:Building dependency tree... 2019-10-17T20:43:15.672 INFO:teuthology.orchestra.run.smithi092.stdout:Reading state information... 2019-10-17T20:43:15.730 INFO:teuthology.orchestra.run.smithi204.stdout:Building dependency tree... 2019-10-17T20:43:15.731 INFO:teuthology.orchestra.run.smithi204.stdout:Reading state information... 2019-10-17T20:43:15.831 INFO:teuthology.orchestra.run.smithi092.stdout:The following additional packages will be installed: 2019-10-17T20:43:15.831 INFO:teuthology.orchestra.run.smithi092.stdout: bridge-utils ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-17T20:43:15.831 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-17T20:43:15.831 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon ceph-osd cgroupfs-mount containerd dns-root-data dnsmasq-base 2019-10-17T20:43:15.832 INFO:teuthology.orchestra.run.smithi092.stdout: docker.io formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-10-17T20:43:15.832 INFO:teuthology.orchestra.run.smithi092.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: python-chardet python-cherrypy3 python-decorator python-dnspython 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: python-fasteners python-formencode python-funcsigs python-html5lib 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: python-httplib2 python-joblib python-jwt python-kubernetes python-logutils 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: python-lxml python-mako python-markupsafe python-monotonic 2019-10-17T20:43:15.833 INFO:teuthology.orchestra.run.smithi092.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-17T20:43:15.834 INFO:teuthology.orchestra.run.smithi092.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-17T20:43:15.835 INFO:teuthology.orchestra.run.smithi092.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-17T20:43:15.835 INFO:teuthology.orchestra.run.smithi092.stdout: python-webtest python-werkzeug python-yaml runc socat ubuntu-fan xmlstarlet 2019-10-17T20:43:15.835 INFO:teuthology.orchestra.run.smithi092.stdout:Suggested packages: 2019-10-17T20:43:15.835 INFO:teuthology.orchestra.run.smithi092.stdout: python-influxdb aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils 2019-10-17T20:43:15.835 INFO:teuthology.orchestra.run.smithi092.stdout: python-attr-doc python-egenix-mxdatetime python-funcsigs-doc python-genshi 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-10-17T20:43:15.836 INFO:teuthology.orchestra.run.smithi092.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-10-17T20:43:15.837 INFO:teuthology.orchestra.run.smithi092.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-10-17T20:43:15.837 INFO:teuthology.orchestra.run.smithi092.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-10-17T20:43:15.837 INFO:teuthology.orchestra.run.smithi092.stdout: python-watchdog python-werkzeug-doc 2019-10-17T20:43:15.881 INFO:teuthology.orchestra.run.smithi204.stdout:The following additional packages will be installed: 2019-10-17T20:43:15.881 INFO:teuthology.orchestra.run.smithi204.stdout: bridge-utils ceph-base ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-17T20:43:15.881 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-17T20:43:15.881 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon ceph-osd cgroupfs-mount containerd dns-root-data dnsmasq-base 2019-10-17T20:43:15.882 INFO:teuthology.orchestra.run.smithi204.stdout: docker.io formencode-i18n jq libjq1 libjs-sphinxdoc libjs-underscore 2019-10-17T20:43:15.882 INFO:teuthology.orchestra.run.smithi204.stdout: libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 libsnappy1v5 libxslt1.1 pigz python-attr python-bcrypt python-bs4 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-ceph-argparse python-ceph-common python-cephfs python-certifi 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-chardet python-cherrypy3 python-decorator python-dnspython 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-fasteners python-formencode python-funcsigs python-html5lib 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-httplib2 python-joblib python-jwt python-kubernetes python-logutils 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-lxml python-mako python-markupsafe python-monotonic 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-oauth2client python-openid python-openssl python-paste 2019-10-17T20:43:15.883 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy python-pastedeploy-tpl python-pastescript python-pecan 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-pluggy python-prettytable python-py python-pyasn1 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyasn1-modules python-pyinotify python-pytest python-rados python-rbd 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-repoze.lru python-requests python-rgw python-routes python-rsa 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-scgi python-scipy python-simplegeneric python-simplejson 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-singledispatch python-sklearn python-sklearn-lib python-sqlalchemy 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-sqlalchemy-ext python-tempita python-uritemplate python-urllib3 2019-10-17T20:43:15.884 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress python-webencodings python-webob python-websocket 2019-10-17T20:43:15.885 INFO:teuthology.orchestra.run.smithi204.stdout: python-webtest python-werkzeug python-yaml runc socat ubuntu-fan xmlstarlet 2019-10-17T20:43:15.885 INFO:teuthology.orchestra.run.smithi204.stdout:Suggested packages: 2019-10-17T20:43:15.885 INFO:teuthology.orchestra.run.smithi204.stdout: python-influxdb aufs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils 2019-10-17T20:43:15.885 INFO:teuthology.orchestra.run.smithi204.stdout: python-attr-doc python-egenix-mxdatetime python-funcsigs-doc python-genshi 2019-10-17T20:43:15.885 INFO:teuthology.orchestra.run.smithi204.stdout: python-lxml-dbg python-lxml-doc python-beaker python-mako-doc 2019-10-17T20:43:15.885 INFO:teuthology.orchestra.run.smithi204.stdout: python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: libapache2-mod-scgi libjs-mochikit python-pastewebkit python-cheetah 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: python-cherrypy python-flup python-pecan-doc subversion python-pytest-xdist 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyinotify-doc python-mock python-socks python-scipy-doc python-dap 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: python-scikits-optimization python-sklearn-doc ipython python-sqlalchemy-doc 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: python-psycopg2 python-pymysql python-fdb python-pymssql python-ntlm 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-pyquery 2019-10-17T20:43:15.886 INFO:teuthology.orchestra.run.smithi204.stdout: python-redis python-pylibmc | python-memcache python-termcolor 2019-10-17T20:43:15.887 INFO:teuthology.orchestra.run.smithi204.stdout: python-watchdog python-werkzeug-doc 2019-10-17T20:43:15.931 INFO:teuthology.orchestra.run.smithi092.stdout:The following NEW packages will be installed: 2019-10-17T20:43:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: bridge-utils ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds 2019-10-17T20:43:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-17T20:43:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-17T20:43:15.932 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon ceph-osd ceph-test cgroupfs-mount containerd dns-root-data 2019-10-17T20:43:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: dnsmasq-base docker.io formencode-i18n jq libcephfs-dev libcephfs2 libjq1 2019-10-17T20:43:15.933 INFO:teuthology.orchestra.run.smithi092.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 liboath0 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: libonig4 librabbitmq4 libradosstriper1 librgw2 libsnappy1v5 libxslt1.1 pigz 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: python-attr python-bcrypt python-bs4 python-ceph python-ceph-argparse 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: python-ceph-common python-cephfs python-certifi python-chardet 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: python-cherrypy3 python-decorator python-dnspython python-fasteners 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-10-17T20:43:15.934 INFO:teuthology.orchestra.run.smithi092.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-10-17T20:43:15.935 INFO:teuthology.orchestra.run.smithi092.stdout: python-openid python-openssl python-paste python-pastedeploy 2019-10-17T20:43:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: python-pastedeploy-tpl python-pastescript python-pecan python-pluggy 2019-10-17T20:43:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: python-prettytable python-py python-pyasn1 python-pyasn1-modules 2019-10-17T20:43:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: python-pyinotify python-pytest python-rados python-rbd python-repoze.lru 2019-10-17T20:43:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: python-requests python-rgw python-routes python-rsa python-scgi python-scipy 2019-10-17T20:43:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-10-17T20:43:15.936 INFO:teuthology.orchestra.run.smithi092.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-10-17T20:43:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-10-17T20:43:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-10-17T20:43:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: radosgw rbd-fuse runc socat ubuntu-fan xmlstarlet 2019-10-17T20:43:15.937 INFO:teuthology.orchestra.run.smithi092.stdout:The following packages will be upgraded: 2019-10-17T20:43:15.937 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 librbd1 2019-10-17T20:43:15.983 INFO:teuthology.orchestra.run.smithi204.stdout:The following NEW packages will be installed: 2019-10-17T20:43:15.984 INFO:teuthology.orchestra.run.smithi204.stdout: bridge-utils ceph ceph-base ceph-common ceph-daemon ceph-fuse ceph-mds 2019-10-17T20:43:15.984 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2019-10-17T20:43:15.984 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mgr-ssh 2019-10-17T20:43:15.984 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon ceph-osd ceph-test cgroupfs-mount containerd dns-root-data 2019-10-17T20:43:15.985 INFO:teuthology.orchestra.run.smithi204.stdout: dnsmasq-base docker.io formencode-i18n jq libcephfs-dev libcephfs2 libjq1 2019-10-17T20:43:15.985 INFO:teuthology.orchestra.run.smithi204.stdout: libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust0 liboath0 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: libonig4 librabbitmq4 libradosstriper1 librgw2 libsnappy1v5 libxslt1.1 pigz 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-attr python-bcrypt python-bs4 python-ceph python-ceph-argparse 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-ceph-common python-cephfs python-certifi python-chardet 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-cherrypy3 python-decorator python-dnspython python-fasteners 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-formencode python-funcsigs python-html5lib python-httplib2 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-joblib python-jwt python-kubernetes python-logutils python-lxml 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-mako python-markupsafe python-monotonic python-oauth2client 2019-10-17T20:43:15.986 INFO:teuthology.orchestra.run.smithi204.stdout: python-openid python-openssl python-paste python-pastedeploy 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-pastedeploy-tpl python-pastescript python-pecan python-pluggy 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-prettytable python-py python-pyasn1 python-pyasn1-modules 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-pyinotify python-pytest python-rados python-rbd python-repoze.lru 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-requests python-rgw python-routes python-rsa python-scgi python-scipy 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-simplegeneric python-simplejson python-singledispatch python-sklearn 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-sklearn-lib python-sqlalchemy python-sqlalchemy-ext python-tempita 2019-10-17T20:43:15.987 INFO:teuthology.orchestra.run.smithi204.stdout: python-uritemplate python-urllib3 python-waitress python-webencodings 2019-10-17T20:43:15.988 INFO:teuthology.orchestra.run.smithi204.stdout: python-webob python-websocket python-webtest python-werkzeug python-yaml 2019-10-17T20:43:15.988 INFO:teuthology.orchestra.run.smithi204.stdout: radosgw rbd-fuse runc socat ubuntu-fan xmlstarlet 2019-10-17T20:43:15.988 INFO:teuthology.orchestra.run.smithi204.stdout:The following packages will be upgraded: 2019-10-17T20:43:15.988 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 librbd1 2019-10-17T20:43:16.085 INFO:teuthology.orchestra.run.smithi092.stdout:2 upgraded, 111 newly installed, 0 to remove and 88 not upgraded. 2019-10-17T20:43:16.085 INFO:teuthology.orchestra.run.smithi092.stdout:Need to get 182 MB of archives. 2019-10-17T20:43:16.086 INFO:teuthology.orchestra.run.smithi092.stdout:After this operation, 867 MB of additional disk space will be used. 2019-10-17T20:43:16.086 INFO:teuthology.orchestra.run.smithi092.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-6144-gf0009d4-1bionic [1,490 kB] 2019-10-17T20:43:16.125 INFO:teuthology.orchestra.run.smithi092.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-10-17T20:43:16.130 INFO:teuthology.orchestra.run.smithi204.stdout:2 upgraded, 111 newly installed, 0 to remove and 88 not upgraded. 2019-10-17T20:43:16.130 INFO:teuthology.orchestra.run.smithi204.stdout:Need to get 182 MB of archives. 2019-10-17T20:43:16.130 INFO:teuthology.orchestra.run.smithi204.stdout:After this operation, 867 MB of additional disk space will be used. 2019-10-17T20:43:16.130 INFO:teuthology.orchestra.run.smithi204.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-6144-gf0009d4-1bionic [1,490 kB] 2019-10-17T20:43:16.176 INFO:teuthology.orchestra.run.smithi204.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-uritemplate all 0.6-1ubuntu1 [5,042 B] 2019-10-17T20:43:16.228 INFO:teuthology.orchestra.run.smithi092.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-6144-gf0009d4-1bionic [2,977 kB] 2019-10-17T20:43:16.286 INFO:teuthology.orchestra.run.smithi092.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB] 2019-10-17T20:43:16.337 INFO:teuthology.orchestra.run.smithi092.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-6144-gf0009d4-1bionic [426 kB] 2019-10-17T20:43:16.351 INFO:teuthology.orchestra.run.smithi092.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-6144-gf0009d4-1bionic [303 kB] 2019-10-17T20:43:16.351 INFO:teuthology.orchestra.run.smithi204.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB] 2019-10-17T20:43:16.361 INFO:teuthology.orchestra.run.smithi092.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-6144-gf0009d4-1bionic [20.2 kB] 2019-10-17T20:43:16.362 INFO:teuthology.orchestra.run.smithi092.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-6144-gf0009d4-1bionic [103 kB] 2019-10-17T20:43:16.366 INFO:teuthology.orchestra.run.smithi092.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-6144-gf0009d4-1bionic [10.4 kB] 2019-10-17T20:43:16.367 INFO:teuthology.orchestra.run.smithi092.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-6144-gf0009d4-1bionic [249 kB] 2019-10-17T20:43:16.373 INFO:teuthology.orchestra.run.smithi092.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-6144-gf0009d4-1bionic [4,494 kB] 2019-10-17T20:43:16.437 INFO:teuthology.orchestra.run.smithi204.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-6144-gf0009d4-1bionic [2,977 kB] 2019-10-17T20:43:16.550 INFO:teuthology.orchestra.run.smithi092.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-6144-gf0009d4-1bionic [98.7 kB] 2019-10-17T20:43:16.552 INFO:teuthology.orchestra.run.smithi092.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-6144-gf0009d4-1bionic [329 kB] 2019-10-17T20:43:16.563 INFO:teuthology.orchestra.run.smithi092.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-6144-gf0009d4-1bionic [17.8 MB] 2019-10-17T20:43:16.626 INFO:teuthology.orchestra.run.smithi092.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB] 2019-10-17T20:43:16.700 INFO:teuthology.orchestra.run.smithi204.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 bridge-utils amd64 1.5-15ubuntu1 [30.1 kB] 2019-10-17T20:43:16.867 INFO:teuthology.orchestra.run.smithi092.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-10-17T20:43:16.970 INFO:teuthology.orchestra.run.smithi204.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2019-10-17T20:43:17.002 INFO:teuthology.orchestra.run.smithi204.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-6144-gf0009d4-1bionic [426 kB] 2019-10-17T20:43:17.007 INFO:teuthology.orchestra.run.smithi204.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-rados amd64 15.0.0-6144-gf0009d4-1bionic [303 kB] 2019-10-17T20:43:17.054 INFO:teuthology.orchestra.run.smithi204.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-argparse all 15.0.0-6144-gf0009d4-1bionic [20.2 kB] 2019-10-17T20:43:17.055 INFO:teuthology.orchestra.run.smithi204.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-cephfs amd64 15.0.0-6144-gf0009d4-1bionic [103 kB] 2019-10-17T20:43:17.106 INFO:teuthology.orchestra.run.smithi204.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph-common all 15.0.0-6144-gf0009d4-1bionic [10.4 kB] 2019-10-17T20:43:17.107 INFO:teuthology.orchestra.run.smithi204.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-rbd amd64 15.0.0-6144-gf0009d4-1bionic [249 kB] 2019-10-17T20:43:17.115 INFO:teuthology.orchestra.run.smithi204.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-6144-gf0009d4-1bionic [4,494 kB] 2019-10-17T20:43:17.287 INFO:teuthology.orchestra.run.smithi092.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-6144-gf0009d4-1bionic [4,974 kB] 2019-10-17T20:43:17.316 INFO:teuthology.orchestra.run.smithi092.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-10-17T20:43:17.439 INFO:teuthology.orchestra.run.smithi204.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-prettytable all 0.7.2-3 [19.6 kB] 2019-10-17T20:43:17.491 INFO:teuthology.orchestra.run.smithi092.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-6144-gf0009d4-1bionic [1,323 kB] 2019-10-17T20:43:17.546 INFO:teuthology.orchestra.run.smithi092.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-6144-gf0009d4-1bionic [5,895 kB] 2019-10-17T20:43:17.557 INFO:teuthology.orchestra.run.smithi092.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-10-17T20:43:17.696 INFO:teuthology.orchestra.run.smithi204.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.1 [33.9 kB] 2019-10-17T20:43:17.777 INFO:teuthology.orchestra.run.smithi092.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-6144-gf0009d4-1bionic [22.4 MB] 2019-10-17T20:43:17.799 INFO:teuthology.orchestra.run.smithi092.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-10-17T20:43:17.925 INFO:teuthology.orchestra.run.smithi204.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-rgw amd64 15.0.0-6144-gf0009d4-1bionic [98.7 kB] 2019-10-17T20:43:17.929 INFO:teuthology.orchestra.run.smithi204.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-6144-gf0009d4-1bionic [329 kB] 2019-10-17T20:43:17.982 INFO:teuthology.orchestra.run.smithi204.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2019-10-17T20:43:17.993 INFO:teuthology.orchestra.run.smithi204.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-6144-gf0009d4-1bionic [17.8 MB] 2019-10-17T20:43:18.040 INFO:teuthology.orchestra.run.smithi092.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-10-17T20:43:18.224 INFO:teuthology.orchestra.run.smithi204.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2019-10-17T20:43:18.480 INFO:teuthology.orchestra.run.smithi092.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-10-17T20:43:18.679 INFO:teuthology.orchestra.run.smithi204.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2019-10-17T20:43:18.692 INFO:teuthology.orchestra.run.smithi092.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-6144-gf0009d4-1bionic [3,784 B] 2019-10-17T20:43:18.692 INFO:teuthology.orchestra.run.smithi092.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-6144-gf0009d4-1bionic [12.5 kB] 2019-10-17T20:43:18.692 INFO:teuthology.orchestra.run.smithi092.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-6144-gf0009d4-1bionic [504 kB] 2019-10-17T20:43:18.703 INFO:teuthology.orchestra.run.smithi092.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-6144-gf0009d4-1bionic [1,822 kB] 2019-10-17T20:43:18.777 INFO:teuthology.orchestra.run.smithi092.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-6144-gf0009d4-1bionic [3,353 kB] 2019-10-17T20:43:18.804 INFO:teuthology.orchestra.run.smithi092.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-10-17T20:43:18.913 INFO:teuthology.orchestra.run.smithi092.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-6144-gf0009d4-1bionic [34.3 kB] 2019-10-17T20:43:18.914 INFO:teuthology.orchestra.run.smithi092.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-6144-gf0009d4-1bionic [1,091 kB] 2019-10-17T20:43:18.963 INFO:teuthology.orchestra.run.smithi092.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-6144-gf0009d4-1bionic [3,896 B] 2019-10-17T20:43:18.964 INFO:teuthology.orchestra.run.smithi092.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-6144-gf0009d4-1bionic [13.6 kB] 2019-10-17T20:43:18.964 INFO:teuthology.orchestra.run.smithi092.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-6144-gf0009d4-1bionic [12.8 kB] 2019-10-17T20:43:18.964 INFO:teuthology.orchestra.run.smithi092.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-6144-gf0009d4-1bionic [35.3 MB] 2019-10-17T20:43:19.008 INFO:teuthology.orchestra.run.smithi204.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bcrypt amd64 3.1.4-2 [29.8 kB] 2019-10-17T20:43:19.051 INFO:teuthology.orchestra.run.smithi092.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-10-17T20:43:19.278 INFO:teuthology.orchestra.run.smithi204.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-cherrypy3 all 8.9.1-2 [160 kB] 2019-10-17T20:43:19.506 INFO:teuthology.orchestra.run.smithi092.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-10-17T20:43:19.747 INFO:teuthology.orchestra.run.smithi092.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-10-17T20:43:19.752 INFO:teuthology.orchestra.run.smithi204.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-jwt all 1.5.3+ds1-1 [15.8 kB] 2019-10-17T20:43:20.001 INFO:teuthology.orchestra.run.smithi204.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openssl all 17.5.0-1ubuntu1 [41.3 kB] 2019-10-17T20:43:20.069 INFO:teuthology.orchestra.run.smithi092.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-10-17T20:43:20.309 INFO:teuthology.orchestra.run.smithi092.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-10-17T20:43:20.329 INFO:teuthology.orchestra.run.smithi204.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-logutils all 0.3.3-5 [16.7 kB] 2019-10-17T20:43:20.398 INFO:teuthology.orchestra.run.smithi092.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-6144-gf0009d4-1bionic [17.3 kB] 2019-10-17T20:43:20.398 INFO:teuthology.orchestra.run.smithi092.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-6144-gf0009d4-1bionic [3,840 B] 2019-10-17T20:43:20.399 INFO:teuthology.orchestra.run.smithi092.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-6144-gf0009d4-1bionic [5,154 kB] 2019-10-17T20:43:20.471 INFO:teuthology.orchestra.run.smithi092.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-10-17T20:43:20.578 INFO:teuthology.orchestra.run.smithi204.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-markupsafe amd64 1.0-1build1 [13.0 kB] 2019-10-17T20:43:20.604 INFO:teuthology.orchestra.run.smithi092.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-6144-gf0009d4-1bionic [61.9 kB] 2019-10-17T20:43:20.730 INFO:teuthology.orchestra.run.smithi204.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-6144-gf0009d4-1bionic [4,974 kB] 2019-10-17T20:43:20.762 INFO:teuthology.orchestra.run.smithi204.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-mako all 1.0.7+ds1-1 [60.1 kB] 2019-10-17T20:43:20.817 INFO:teuthology.orchestra.run.smithi092.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-10-17T20:43:20.956 INFO:teuthology.orchestra.run.smithi204.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-6144-gf0009d4-1bionic [1,323 kB] 2019-10-17T20:43:20.978 INFO:teuthology.orchestra.run.smithi092.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-10-17T20:43:21.014 INFO:teuthology.orchestra.run.smithi204.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-6144-gf0009d4-1bionic [5,895 kB] 2019-10-17T20:43:21.129 INFO:teuthology.orchestra.run.smithi204.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplegeneric all 0.8.1-1 [11.5 kB] 2019-10-17T20:43:21.139 INFO:teuthology.orchestra.run.smithi092.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-10-17T20:43:21.280 INFO:teuthology.orchestra.run.smithi204.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-6144-gf0009d4-1bionic [22.4 MB] 2019-10-17T20:43:21.325 INFO:teuthology.orchestra.run.smithi204.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-singledispatch all 3.4.0.3-2 [9,848 B] 2019-10-17T20:43:21.494 INFO:teuthology.orchestra.run.smithi092.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-10-17T20:43:21.512 INFO:teuthology.orchestra.run.smithi204.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-simplejson amd64 3.13.2-1 [61.2 kB] 2019-10-17T20:43:21.844 INFO:teuthology.orchestra.run.smithi092.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-10-17T20:43:21.873 INFO:teuthology.orchestra.run.smithi204.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2019-10-17T20:43:22.194 INFO:teuthology.orchestra.run.smithi092.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-10-17T20:43:22.247 INFO:teuthology.orchestra.run.smithi204.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-bs4 all 4.6.0-1 [67.9 kB] 2019-10-17T20:43:22.293 INFO:teuthology.orchestra.run.smithi204.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-6144-gf0009d4-1bionic [3,784 B] 2019-10-17T20:43:22.293 INFO:teuthology.orchestra.run.smithi204.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-daemon amd64 15.0.0-6144-gf0009d4-1bionic [12.5 kB] 2019-10-17T20:43:22.294 INFO:teuthology.orchestra.run.smithi204.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-6144-gf0009d4-1bionic [504 kB] 2019-10-17T20:43:22.314 INFO:teuthology.orchestra.run.smithi204.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-6144-gf0009d4-1bionic [1,822 kB] 2019-10-17T20:43:22.395 INFO:teuthology.orchestra.run.smithi204.stdout:Get:42 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-6144-gf0009d4-1bionic [3,353 kB] 2019-10-17T20:43:22.529 INFO:teuthology.orchestra.run.smithi092.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-10-17T20:43:22.546 INFO:teuthology.orchestra.run.smithi204.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-6144-gf0009d4-1bionic [34.3 kB] 2019-10-17T20:43:22.549 INFO:teuthology.orchestra.run.smithi204.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-6144-gf0009d4-1bionic [1,091 kB] 2019-10-17T20:43:22.599 INFO:teuthology.orchestra.run.smithi204.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-6144-gf0009d4-1bionic [3,896 B] 2019-10-17T20:43:22.600 INFO:teuthology.orchestra.run.smithi204.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-6144-gf0009d4-1bionic [13.6 kB] 2019-10-17T20:43:22.600 INFO:teuthology.orchestra.run.smithi204.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-ssh all 15.0.0-6144-gf0009d4-1bionic [12.8 kB] 2019-10-17T20:43:22.601 INFO:teuthology.orchestra.run.smithi204.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-6144-gf0009d4-1bionic [35.3 MB] 2019-10-17T20:43:22.613 INFO:teuthology.orchestra.run.smithi204.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-waitress all 1.0.1-1 [53.3 kB] 2019-10-17T20:43:22.893 INFO:teuthology.orchestra.run.smithi092.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-10-17T20:43:22.969 INFO:teuthology.orchestra.run.smithi204.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-dnspython all 1.15.0-1 [84.9 kB] 2019-10-17T20:43:23.054 INFO:teuthology.orchestra.run.smithi092.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-10-17T20:43:23.359 INFO:teuthology.orchestra.run.smithi204.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 formencode-i18n all 1.3.0-0ubuntu5 [3,382 B] 2019-10-17T20:43:23.409 INFO:teuthology.orchestra.run.smithi092.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-10-17T20:43:23.529 INFO:teuthology.orchestra.run.smithi204.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-formencode all 1.3.0-0ubuntu5 [69.1 kB] 2019-10-17T20:43:23.650 INFO:teuthology.orchestra.run.smithi092.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-10-17T20:43:23.906 INFO:teuthology.orchestra.run.smithi204.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-tempita all 0.5.2-2 [13.9 kB] 2019-10-17T20:43:24.149 INFO:teuthology.orchestra.run.smithi204.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2019-10-17T20:43:24.174 INFO:teuthology.orchestra.run.smithi204.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-6144-gf0009d4-1bionic [17.3 kB] 2019-10-17T20:43:24.174 INFO:teuthology.orchestra.run.smithi204.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 python-ceph amd64 15.0.0-6144-gf0009d4-1bionic [3,840 B] 2019-10-17T20:43:24.174 INFO:teuthology.orchestra.run.smithi204.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-6144-gf0009d4-1bionic [5,154 kB] 2019-10-17T20:43:24.229 INFO:teuthology.orchestra.run.smithi092.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-10-17T20:43:24.389 INFO:teuthology.orchestra.run.smithi204.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-sjust-testing2/f0009d4765d23f6e6a0d55d83c1e9f63c45550d5/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-6144-gf0009d4-1bionic [61.9 kB] 2019-10-17T20:43:24.391 INFO:teuthology.orchestra.run.smithi092.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-10-17T20:43:24.632 INFO:teuthology.orchestra.run.smithi092.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-10-17T20:43:24.744 INFO:teuthology.orchestra.run.smithi204.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2019-10-17T20:43:24.875 INFO:teuthology.orchestra.run.smithi092.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-10-17T20:43:24.912 INFO:teuthology.orchestra.run.smithi204.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy all 1.5.2-4 [25.6 kB] 2019-10-17T20:43:25.194 INFO:teuthology.orchestra.run.smithi204.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webtest all 2.0.28-1ubuntu1 [27.8 kB] 2019-10-17T20:43:25.223 INFO:teuthology.orchestra.run.smithi092.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-10-17T20:43:25.464 INFO:teuthology.orchestra.run.smithi204.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pecan all 1.2.1-2 [86.1 kB] 2019-10-17T20:43:25.657 INFO:teuthology.orchestra.run.smithi092.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-10-17T20:43:25.859 INFO:teuthology.orchestra.run.smithi204.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-certifi all 2018.1.18-2 [144 kB] 2019-10-17T20:43:26.006 INFO:teuthology.orchestra.run.smithi092.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-10-17T20:43:26.321 INFO:teuthology.orchestra.run.smithi204.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-chardet all 3.0.4-1 [80.3 kB] 2019-10-17T20:43:26.354 INFO:teuthology.orchestra.run.smithi092.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-10-17T20:43:26.697 INFO:teuthology.orchestra.run.smithi092.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-10-17T20:43:26.706 INFO:teuthology.orchestra.run.smithi204.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-urllib3 all 1.22-1ubuntu0.18.04.1 [85.9 kB] 2019-10-17T20:43:27.100 INFO:teuthology.orchestra.run.smithi204.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-requests all 2.18.4-2ubuntu0.1 [58.5 kB] 2019-10-17T20:43:27.158 INFO:teuthology.orchestra.run.smithi092.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 [1,903 kB] 2019-10-17T20:43:27.453 INFO:teuthology.orchestra.run.smithi204.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2019-10-17T20:43:27.904 INFO:teuthology.orchestra.run.smithi092.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.2.6-0ubuntu1~18.04.2 [19.4 MB] 2019-10-17T20:43:27.935 INFO:teuthology.orchestra.run.smithi204.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 runc amd64 1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2 [1,903 kB] 2019-10-17T20:43:28.693 INFO:teuthology.orchestra.run.smithi204.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 containerd amd64 1.2.6-0ubuntu1~18.04.2 [19.4 MB] 2019-10-17T20:43:29.157 INFO:teuthology.orchestra.run.smithi092.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 18.09.7-0ubuntu1~18.04.4 [30.7 MB] 2019-10-17T20:43:29.966 INFO:teuthology.orchestra.run.smithi204.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 docker.io amd64 18.09.7-0ubuntu1~18.04.4 [30.7 MB] 2019-10-17T20:43:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-10-17T20:43:30.896 INFO:teuthology.orchestra.run.smithi092.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-10-17T20:43:31.218 INFO:teuthology.orchestra.run.smithi092.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-10-17T20:43:31.562 INFO:teuthology.orchestra.run.smithi204.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-repoze.lru all 0.7-1 [11.9 kB] 2019-10-17T20:43:31.567 INFO:teuthology.orchestra.run.smithi092.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-10-17T20:43:31.758 INFO:teuthology.orchestra.run.smithi204.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-underscore all 1.8.3~dfsg-1 [59.9 kB] 2019-10-17T20:43:31.934 INFO:teuthology.orchestra.run.smithi092.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-10-17T20:43:32.096 INFO:teuthology.orchestra.run.smithi092.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-10-17T20:43:32.113 INFO:teuthology.orchestra.run.smithi204.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 libjs-sphinxdoc all 1.6.7-1ubuntu1 [85.6 kB] 2019-10-17T20:43:32.502 INFO:teuthology.orchestra.run.smithi204.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-routes all 2.4.1-1 [88.3 kB] 2019-10-17T20:43:32.895 INFO:teuthology.orchestra.run.smithi204.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-decorator all 4.1.2-1 [9,300 B] 2019-10-17T20:43:33.073 INFO:teuthology.orchestra.run.smithi204.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-scipy amd64 0.19.1-2ubuntu1 [9,689 kB] 2019-10-17T20:43:33.076 INFO:teuthology.orchestra.run.smithi092.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-10-17T20:43:33.819 INFO:teuthology.orchestra.run.smithi092.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-10-17T20:43:34.062 INFO:teuthology.orchestra.run.smithi204.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn-lib amd64 0.19.1-3 [1,521 kB] 2019-10-17T20:43:34.248 INFO:teuthology.orchestra.run.smithi092.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-10-17T20:43:34.784 INFO:teuthology.orchestra.run.smithi204.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-joblib all 0.11-1 [112 kB] 2019-10-17T20:43:34.963 INFO:teuthology.orchestra.run.smithi092.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-10-17T20:43:35.125 INFO:teuthology.orchestra.run.smithi092.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-10-17T20:43:35.204 INFO:teuthology.orchestra.run.smithi204.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-sklearn all 0.19.1-3 [1,373 kB] 2019-10-17T20:43:35.286 INFO:teuthology.orchestra.run.smithi092.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-10-17T20:43:35.528 INFO:teuthology.orchestra.run.smithi092.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-10-17T20:43:35.850 INFO:teuthology.orchestra.run.smithi092.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-10-17T20:43:35.922 INFO:teuthology.orchestra.run.smithi204.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-monotonic all 1.1-2 [5,262 B] 2019-10-17T20:43:36.091 INFO:teuthology.orchestra.run.smithi092.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-10-17T20:43:36.101 INFO:teuthology.orchestra.run.smithi204.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-fasteners all 0.12.0-3 [13.4 kB] 2019-10-17T20:43:36.304 INFO:teuthology.orchestra.run.smithi204.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [34.7 kB] 2019-10-17T20:43:36.333 INFO:teuthology.orchestra.run.smithi092.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-10-17T20:43:36.585 INFO:teuthology.orchestra.run.smithi204.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1 all 0.4.2-3 [46.7 kB] 2019-10-17T20:43:36.921 INFO:teuthology.orchestra.run.smithi204.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyasn1-modules all 0.2.1-0.2 [32.8 kB] 2019-10-17T20:43:36.966 INFO:teuthology.orchestra.run.smithi092.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-10-17T20:43:37.197 INFO:teuthology.orchestra.run.smithi204.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-rsa all 3.4.2-1 [28.9 kB] 2019-10-17T20:43:37.314 INFO:teuthology.orchestra.run.smithi092.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-10-17T20:43:37.485 INFO:teuthology.orchestra.run.smithi204.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy all 1.1.11+ds1-1ubuntu1 [682 kB] 2019-10-17T20:43:37.555 INFO:teuthology.orchestra.run.smithi092.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-10-17T20:43:37.976 INFO:teuthology.orchestra.run.smithi092.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-10-17T20:43:38.120 INFO:teuthology.orchestra.run.smithi204.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-oauth2client all 4.1.2-3 [61.4 kB] 2019-10-17T20:43:38.454 INFO:teuthology.orchestra.run.smithi092.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-10-17T20:43:38.478 INFO:teuthology.orchestra.run.smithi204.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2019-10-17T20:43:38.779 INFO:teuthology.orchestra.run.smithi204.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-yaml amd64 3.12-1build2 [115 kB] 2019-10-17T20:43:38.884 INFO:teuthology.orchestra.run.smithi092.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-10-17T20:43:39.213 INFO:teuthology.orchestra.run.smithi204.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2019-10-17T20:43:39.298 INFO:teuthology.orchestra.run.smithi092.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-10-17T20:43:39.623 INFO:teuthology.orchestra.run.smithi092.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-10-17T20:43:39.691 INFO:teuthology.orchestra.run.smithi204.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2019-10-17T20:43:40.132 INFO:teuthology.orchestra.run.smithi204.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2019-10-17T20:43:40.159 INFO:teuthology.orchestra.run.smithi092.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.1 [150 kB] 2019-10-17T20:43:40.566 INFO:teuthology.orchestra.run.smithi204.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2019-10-17T20:43:40.600 INFO:teuthology.orchestra.run.smithi092.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-10-17T20:43:40.903 INFO:teuthology.orchestra.run.smithi204.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2019-10-17T20:43:41.154 INFO:teuthology.orchestra.run.smithi092.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B] 2019-10-17T20:43:41.316 INFO:teuthology.orchestra.run.smithi092.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5,360 B] 2019-10-17T20:43:41.451 INFO:teuthology.orchestra.run.smithi204.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.1 [150 kB] 2019-10-17T20:43:41.478 INFO:teuthology.orchestra.run.smithi092.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/main amd64 dnsmasq-base amd64 2.79-1 [306 kB] 2019-10-17T20:43:41.917 INFO:teuthology.orchestra.run.smithi204.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2019-10-17T20:43:42.023 INFO:teuthology.orchestra.run.smithi092.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-10-17T20:43:42.264 INFO:teuthology.orchestra.run.smithi092.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-10-17T20:43:42.426 INFO:teuthology.orchestra.run.smithi092.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-10-17T20:43:42.487 INFO:teuthology.orchestra.run.smithi204.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B] 2019-10-17T20:43:42.588 INFO:teuthology.orchestra.run.smithi092.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-10-17T20:43:42.661 INFO:teuthology.orchestra.run.smithi204.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 dns-root-data all 2018013001 [5,360 B] 2019-10-17T20:43:42.834 INFO:teuthology.orchestra.run.smithi204.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/main amd64 dnsmasq-base amd64 2.79-1 [306 kB] 2019-10-17T20:43:42.932 INFO:teuthology.orchestra.run.smithi092.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-10-17T20:43:43.385 INFO:teuthology.orchestra.run.smithi204.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2019-10-17T20:43:43.610 INFO:teuthology.orchestra.run.smithi092.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-10-17T20:43:43.654 INFO:teuthology.orchestra.run.smithi204.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2019-10-17T20:43:43.843 INFO:teuthology.orchestra.run.smithi204.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-webencodings all 0.5-2 [10.3 kB] 2019-10-17T20:43:44.016 INFO:teuthology.orchestra.run.smithi092.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-10-17T20:43:44.036 INFO:teuthology.orchestra.run.smithi204.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-html5lib all 0.999999999-1 [83.6 kB] 2019-10-17T20:43:44.356 INFO:teuthology.orchestra.run.smithi092.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-10-17T20:43:44.442 INFO:teuthology.orchestra.run.smithi204.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python-lxml amd64 4.2.1-1ubuntu0.1 [1,075 kB] 2019-10-17T20:43:44.517 INFO:teuthology.orchestra.run.smithi092.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-10-17T20:43:44.861 INFO:teuthology.orchestra.run.smithi092.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-10-17T20:43:45.103 INFO:teuthology.orchestra.run.smithi092.stdout:Get:110 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-10-17T20:43:45.123 INFO:teuthology.orchestra.run.smithi204.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-openid all 2.2.5-6 [97.8 kB] 2019-10-17T20:43:45.533 INFO:teuthology.orchestra.run.smithi204.stdout:Get:106 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastescript all 2.0.2-2 [54.9 kB] 2019-10-17T20:43:45.536 INFO:teuthology.orchestra.run.smithi092.stdout:Get:111 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-10-17T20:43:45.777 INFO:teuthology.orchestra.run.smithi092.stdout:Get:112 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-10-17T20:43:45.877 INFO:teuthology.orchestra.run.smithi204.stdout:Get:107 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2019-10-17T20:43:45.939 INFO:teuthology.orchestra.run.smithi092.stdout:Get:113 http://archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB] 2019-10-17T20:43:46.066 INFO:teuthology.orchestra.run.smithi204.stdout:Get:108 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2019-10-17T20:43:46.441 INFO:teuthology.orchestra.run.smithi204.stdout:Get:109 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pyinotify all 0.9.6-1 [24.6 kB] 2019-10-17T20:43:46.649 INFO:teuthology.orchestra.run.smithi092.stdout:Preconfiguring packages ... 2019-10-17T20:43:46.710 INFO:teuthology.orchestra.run.smithi204.stdout:Get:110 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2019-10-17T20:43:46.807 INFO:teuthology.orchestra.run.smithi092.stdout:Fetched 182 MB in 30s (6,058 kB/s) 2019-10-17T20:43:46.927 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-uritemplate. 2019-10-17T20:43:46.975 INFO:teuthology.orchestra.run.smithi092.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 136808 files and directories currently installed.) 2019-10-17T20:43:46.979 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-10-17T20:43:47.009 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-10-17T20:43:47.169 INFO:teuthology.orchestra.run.smithi204.stdout:Get:111 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-scgi amd64 1.13-1.1build1 [18.4 kB] 2019-10-17T20:43:47.228 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package pigz. 2019-10-17T20:43:47.242 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../001-pigz_2.4-1_amd64.deb ... 2019-10-17T20:43:47.286 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking pigz (2.4-1) ... 2019-10-17T20:43:47.417 INFO:teuthology.orchestra.run.smithi204.stdout:Get:112 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-sqlalchemy-ext amd64 1.1.11+ds1-1ubuntu1 [13.4 kB] 2019-10-17T20:43:47.615 INFO:teuthology.orchestra.run.smithi204.stdout:Get:113 http://archive.ubuntu.com/ubuntu bionic/main amd64 ubuntu-fan all 0.12.10 [34.7 kB] 2019-10-17T20:43:47.698 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package bridge-utils. 2019-10-17T20:43:47.706 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../002-bridge-utils_1.5-15ubuntu1_amd64.deb ... 2019-10-17T20:43:47.748 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking bridge-utils (1.5-15ubuntu1) ... 2019-10-17T20:43:48.043 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-10-17T20:43:48.056 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../003-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-10-17T20:43:48.119 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-17T20:43:48.371 INFO:teuthology.orchestra.run.smithi204.stdout:Preconfiguring packages ... 2019-10-17T20:43:48.545 INFO:teuthology.orchestra.run.smithi204.stdout:Fetched 182 MB in 32s (5,740 kB/s) 2019-10-17T20:43:49.111 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../004-librbd1_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:49.248 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librbd1 (15.0.0-6144-gf0009d4-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-17T20:43:49.317 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-uritemplate. 2019-10-17T20:43:49.367 INFO:teuthology.orchestra.run.smithi204.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 136808 files and directories currently installed.) 2019-10-17T20:43:49.371 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../000-python-uritemplate_0.6-1ubuntu1_all.deb ... 2019-10-17T20:43:49.417 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-uritemplate (0.6-1ubuntu1) ... 2019-10-17T20:43:49.635 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package pigz. 2019-10-17T20:43:49.649 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../001-pigz_2.4-1_amd64.deb ... 2019-10-17T20:43:49.685 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking pigz (2.4-1) ... 2019-10-17T20:43:49.764 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../005-librados2_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:49.886 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librados2 (15.0.0-6144-gf0009d4-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-17T20:43:50.038 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package bridge-utils. 2019-10-17T20:43:50.053 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../002-bridge-utils_1.5-15ubuntu1_amd64.deb ... 2019-10-17T20:43:50.088 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking bridge-utils (1.5-15ubuntu1) ... 2019-10-17T20:43:50.417 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2019-10-17T20:43:50.431 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../003-liblttng-ust0_2.10.1-1_amd64.deb ... 2019-10-17T20:43:50.481 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libcephfs2. 2019-10-17T20:43:50.494 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../006-libcephfs2_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:50.500 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-17T20:43:50.539 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcephfs2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:50.795 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../004-librbd1_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:50.842 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-rados. 2019-10-17T20:43:50.855 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../007-python-rados_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:50.892 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-rados (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:50.896 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librbd1 (15.0.0-6144-gf0009d4-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-17T20:43:51.178 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-ceph-argparse. 2019-10-17T20:43:51.191 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../008-python-ceph-argparse_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:43:51.236 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-ceph-argparse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:51.413 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../005-librados2_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:51.455 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-cephfs. 2019-10-17T20:43:51.468 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../009-python-cephfs_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:51.497 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-cephfs (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:51.534 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librados2 (15.0.0-6144-gf0009d4-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2019-10-17T20:43:51.783 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-ceph-common. 2019-10-17T20:43:51.797 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../010-python-ceph-common_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:43:51.841 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:52.096 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs2. 2019-10-17T20:43:52.109 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../006-libcephfs2_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:52.146 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:52.161 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-prettytable. 2019-10-17T20:43:52.175 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../011-python-prettytable_0.7.2-3_all.deb ... 2019-10-17T20:43:52.219 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-10-17T20:43:52.415 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rados. 2019-10-17T20:43:52.428 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../007-python-rados_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:52.463 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-rbd. 2019-10-17T20:43:52.465 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rados (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:52.478 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../012-python-rbd_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:52.513 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-rbd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:52.759 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-ceph-argparse. 2019-10-17T20:43:52.774 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../008-python-ceph-argparse_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:43:52.817 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-ceph-argparse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:52.867 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-10-17T20:43:52.879 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../013-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-10-17T20:43:52.916 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-17T20:43:53.095 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-cephfs. 2019-10-17T20:43:53.109 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../009-python-cephfs_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:53.137 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-cephfs (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:53.229 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package librgw2. 2019-10-17T20:43:53.242 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../014-librgw2_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:53.286 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking librgw2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:53.356 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-ceph-common. 2019-10-17T20:43:53.371 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../010-python-ceph-common_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:43:53.417 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:53.701 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-prettytable. 2019-10-17T20:43:53.716 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../011-python-prettytable_0.7.2-3_all.deb ... 2019-10-17T20:43:53.784 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-prettytable (0.7.2-3) ... 2019-10-17T20:43:53.905 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-rgw. 2019-10-17T20:43:53.918 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../015-python-rgw_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:53.955 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-rgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:54.006 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rbd. 2019-10-17T20:43:54.019 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../012-python-rbd_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:54.047 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rbd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:54.224 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-10-17T20:43:54.237 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-10-17T20:43:54.275 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-17T20:43:54.425 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librabbitmq4:amd64. 2019-10-17T20:43:54.439 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../013-librabbitmq4_0.8.0-1ubuntu0.18.04.1_amd64.deb ... 2019-10-17T20:43:54.483 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-17T20:43:54.577 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-10-17T20:43:54.591 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2019-10-17T20:43:54.635 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-10-17T20:43:54.778 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package librgw2. 2019-10-17T20:43:54.793 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../014-librgw2_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:54.836 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking librgw2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:55.013 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package liboath0. 2019-10-17T20:43:55.026 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2019-10-17T20:43:55.063 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-10-17T20:43:55.324 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libradosstriper1. 2019-10-17T20:43:55.340 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../019-libradosstriper1_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:55.374 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libradosstriper1 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:55.431 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rgw. 2019-10-17T20:43:55.444 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../015-python-rgw_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:55.472 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:55.791 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2019-10-17T20:43:55.806 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2019-10-17T20:43:55.811 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-common. 2019-10-17T20:43:55.824 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../020-ceph-common_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:55.850 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-17T20:43:55.886 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:56.153 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2019-10-17T20:43:56.167 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2019-10-17T20:43:56.211 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2019-10-17T20:43:56.548 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package liboath0. 2019-10-17T20:43:56.561 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2019-10-17T20:43:56.598 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking liboath0 (2.6.1-1) ... 2019-10-17T20:43:56.867 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libradosstriper1. 2019-10-17T20:43:56.881 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../019-libradosstriper1_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:56.925 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libradosstriper1 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:57.355 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-common. 2019-10-17T20:43:57.369 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../020-ceph-common_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:57.412 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:57.474 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-base. 2019-10-17T20:43:57.490 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../021-ceph-base_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:57.533 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-base (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:58.379 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-bcrypt. 2019-10-17T20:43:58.394 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../022-python-bcrypt_3.1.4-2_amd64.deb ... 2019-10-17T20:43:58.437 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-10-17T20:43:58.724 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-cherrypy3. 2019-10-17T20:43:58.739 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../023-python-cherrypy3_8.9.1-2_all.deb ... 2019-10-17T20:43:58.774 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-10-17T20:43:59.093 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-base. 2019-10-17T20:43:59.107 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../021-ceph-base_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:43:59.110 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-jwt. 2019-10-17T20:43:59.125 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../024-python-jwt_1.5.3+ds1-1_all.deb ... 2019-10-17T20:43:59.160 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-10-17T20:43:59.161 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-base (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:43:59.480 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-openssl. 2019-10-17T20:43:59.494 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../025-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-10-17T20:43:59.538 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-10-17T20:43:59.866 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-logutils. 2019-10-17T20:43:59.881 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../026-python-logutils_0.3.3-5_all.deb ... 2019-10-17T20:43:59.925 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-10-17T20:44:00.115 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-bcrypt. 2019-10-17T20:44:00.129 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../022-python-bcrypt_3.1.4-2_amd64.deb ... 2019-10-17T20:44:00.173 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-bcrypt (3.1.4-2) ... 2019-10-17T20:44:00.227 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-markupsafe. 2019-10-17T20:44:00.242 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../027-python-markupsafe_1.0-1build1_amd64.deb ... 2019-10-17T20:44:00.286 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-10-17T20:44:00.443 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-cherrypy3. 2019-10-17T20:44:00.458 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../023-python-cherrypy3_8.9.1-2_all.deb ... 2019-10-17T20:44:00.501 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-cherrypy3 (8.9.1-2) ... 2019-10-17T20:44:00.522 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-mako. 2019-10-17T20:44:00.537 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../028-python-mako_1.0.7+ds1-1_all.deb ... 2019-10-17T20:44:00.572 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-10-17T20:44:00.863 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-jwt. 2019-10-17T20:44:00.876 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../024-python-jwt_1.5.3+ds1-1_all.deb ... 2019-10-17T20:44:00.921 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-jwt (1.5.3+ds1-1) ... 2019-10-17T20:44:00.959 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-simplegeneric. 2019-10-17T20:44:00.974 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../029-python-simplegeneric_0.8.1-1_all.deb ... 2019-10-17T20:44:01.017 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-10-17T20:44:01.233 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-openssl. 2019-10-17T20:44:01.246 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../025-python-openssl_17.5.0-1ubuntu1_all.deb ... 2019-10-17T20:44:01.283 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-openssl (17.5.0-1ubuntu1) ... 2019-10-17T20:44:01.362 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-singledispatch. 2019-10-17T20:44:01.375 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../030-python-singledispatch_3.4.0.3-2_all.deb ... 2019-10-17T20:44:01.412 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-10-17T20:44:01.602 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-logutils. 2019-10-17T20:44:01.618 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../026-python-logutils_0.3.3-5_all.deb ... 2019-10-17T20:44:01.661 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-logutils (0.3.3-5) ... 2019-10-17T20:44:01.706 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-simplejson. 2019-10-17T20:44:01.722 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../031-python-simplejson_3.13.2-1_amd64.deb ... 2019-10-17T20:44:01.756 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-10-17T20:44:01.989 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-markupsafe. 2019-10-17T20:44:02.003 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../027-python-markupsafe_1.0-1build1_amd64.deb ... 2019-10-17T20:44:02.047 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-markupsafe (1.0-1build1) ... 2019-10-17T20:44:02.134 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-webob. 2019-10-17T20:44:02.149 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../032-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-10-17T20:44:02.193 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-10-17T20:44:02.333 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-mako. 2019-10-17T20:44:02.349 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../028-python-mako_1.0.7+ds1-1_all.deb ... 2019-10-17T20:44:02.392 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-mako (1.0.7+ds1-1) ... 2019-10-17T20:44:02.546 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-bs4. 2019-10-17T20:44:02.559 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../033-python-bs4_4.6.0-1_all.deb ... 2019-10-17T20:44:02.596 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-10-17T20:44:02.728 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-simplegeneric. 2019-10-17T20:44:02.743 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../029-python-simplegeneric_0.8.1-1_all.deb ... 2019-10-17T20:44:02.787 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-simplegeneric (0.8.1-1) ... 2019-10-17T20:44:02.926 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-waitress. 2019-10-17T20:44:02.945 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../034-python-waitress_1.0.1-1_all.deb ... 2019-10-17T20:44:02.987 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-10-17T20:44:03.098 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-singledispatch. 2019-10-17T20:44:03.116 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../030-python-singledispatch_3.4.0.3-2_all.deb ... 2019-10-17T20:44:03.164 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-singledispatch (3.4.0.3-2) ... 2019-10-17T20:44:03.349 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-dnspython. 2019-10-17T20:44:03.365 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../035-python-dnspython_1.15.0-1_all.deb ... 2019-10-17T20:44:03.408 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-10-17T20:44:03.525 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-simplejson. 2019-10-17T20:44:03.540 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../031-python-simplejson_3.13.2-1_amd64.deb ... 2019-10-17T20:44:03.584 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-simplejson (3.13.2-1) ... 2019-10-17T20:44:03.678 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package formencode-i18n. 2019-10-17T20:44:03.695 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../036-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-10-17T20:44:03.737 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-17T20:44:03.921 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-webob. 2019-10-17T20:44:03.935 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../032-python-webob_1%3a1.7.3-2fakesync1_all.deb ... 2019-10-17T20:44:03.978 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-webob (1:1.7.3-2fakesync1) ... 2019-10-17T20:44:04.031 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-formencode. 2019-10-17T20:44:04.045 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../037-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-10-17T20:44:04.089 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-10-17T20:44:04.282 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-bs4. 2019-10-17T20:44:04.297 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../033-python-bs4_4.6.0-1_all.deb ... 2019-10-17T20:44:04.340 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-bs4 (4.6.0-1) ... 2019-10-17T20:44:04.450 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-tempita. 2019-10-17T20:44:04.463 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../038-python-tempita_0.5.2-2_all.deb ... 2019-10-17T20:44:04.500 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-10-17T20:44:04.668 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-waitress. 2019-10-17T20:44:04.682 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../034-python-waitress_1.0.1-1_all.deb ... 2019-10-17T20:44:04.727 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-waitress (1.0.1-1) ... 2019-10-17T20:44:04.820 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-paste. 2019-10-17T20:44:04.834 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../039-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-10-17T20:44:04.886 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-17T20:44:05.060 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-dnspython. 2019-10-17T20:44:05.075 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../035-python-dnspython_1.15.0-1_all.deb ... 2019-10-17T20:44:05.120 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-dnspython (1.15.0-1) ... 2019-10-17T20:44:05.198 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-10-17T20:44:05.211 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-10-17T20:44:05.248 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-10-17T20:44:05.438 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package formencode-i18n. 2019-10-17T20:44:05.452 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../036-formencode-i18n_1.3.0-0ubuntu5_all.deb ... 2019-10-17T20:44:05.491 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-17T20:44:05.575 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pastedeploy. 2019-10-17T20:44:05.590 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../041-python-pastedeploy_1.5.2-4_all.deb ... 2019-10-17T20:44:05.642 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-10-17T20:44:05.809 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-formencode. 2019-10-17T20:44:05.823 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../037-python-formencode_1.3.0-0ubuntu5_all.deb ... 2019-10-17T20:44:05.867 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-formencode (1.3.0-0ubuntu5) ... 2019-10-17T20:44:05.978 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-webtest. 2019-10-17T20:44:05.993 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../042-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-10-17T20:44:06.037 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-10-17T20:44:06.204 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-tempita. 2019-10-17T20:44:06.219 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../038-python-tempita_0.5.2-2_all.deb ... 2019-10-17T20:44:06.262 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-tempita (0.5.2-2) ... 2019-10-17T20:44:06.381 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pecan. 2019-10-17T20:44:06.398 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../043-python-pecan_1.2.1-2_all.deb ... 2019-10-17T20:44:06.442 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-10-17T20:44:06.583 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-paste. 2019-10-17T20:44:06.597 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../039-python-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2019-10-17T20:44:06.641 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-17T20:44:06.819 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-certifi. 2019-10-17T20:44:06.834 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../044-python-certifi_2018.1.18-2_all.deb ... 2019-10-17T20:44:06.877 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-10-17T20:44:06.953 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2019-10-17T20:44:06.967 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2019-10-17T20:44:07.011 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2019-10-17T20:44:07.205 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-chardet. 2019-10-17T20:44:07.219 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../045-python-chardet_3.0.4-1_all.deb ... 2019-10-17T20:44:07.264 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-10-17T20:44:07.314 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastedeploy. 2019-10-17T20:44:07.331 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../041-python-pastedeploy_1.5.2-4_all.deb ... 2019-10-17T20:44:07.372 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastedeploy (1.5.2-4) ... 2019-10-17T20:44:07.583 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-urllib3. 2019-10-17T20:44:07.598 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../046-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-10-17T20:44:07.642 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-17T20:44:07.695 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-webtest. 2019-10-17T20:44:07.709 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../042-python-webtest_2.0.28-1ubuntu1_all.deb ... 2019-10-17T20:44:07.751 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-webtest (2.0.28-1ubuntu1) ... 2019-10-17T20:44:08.053 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-requests. 2019-10-17T20:44:08.067 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../047-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-10-17T20:44:08.095 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pecan. 2019-10-17T20:44:08.111 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../043-python-pecan_1.2.1-2_all.deb ... 2019-10-17T20:44:08.120 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-10-17T20:44:08.157 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pecan (1.2.1-2) ... 2019-10-17T20:44:08.439 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-werkzeug. 2019-10-17T20:44:08.455 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../048-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-10-17T20:44:08.497 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-17T20:44:08.517 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-certifi. 2019-10-17T20:44:08.533 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../044-python-certifi_2018.1.18-2_all.deb ... 2019-10-17T20:44:08.575 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-certifi (2018.1.18-2) ... 2019-10-17T20:44:08.859 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr. 2019-10-17T20:44:08.874 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../049-ceph-mgr_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:08.917 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:08.941 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-chardet. 2019-10-17T20:44:08.954 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../045-python-chardet_3.0.4-1_all.deb ... 2019-10-17T20:44:08.995 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-chardet (3.0.4-1) ... 2019-10-17T20:44:09.397 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mon. 2019-10-17T20:44:09.410 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../050-ceph-mon_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:09.455 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:09.715 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-urllib3. 2019-10-17T20:44:09.730 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../046-python-urllib3_1.22-1ubuntu0.18.04.1_all.deb ... 2019-10-17T20:44:09.773 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-17T20:44:10.102 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-requests. 2019-10-17T20:44:10.116 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../047-python-requests_2.18.4-2ubuntu0.1_all.deb ... 2019-10-17T20:44:10.161 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-requests (2.18.4-2ubuntu0.1) ... 2019-10-17T20:44:10.350 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-osd. 2019-10-17T20:44:10.363 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../051-ceph-osd_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:10.400 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-osd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:10.472 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-werkzeug. 2019-10-17T20:44:10.488 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../048-python-werkzeug_0.14.1+dfsg1-1_all.deb ... 2019-10-17T20:44:10.529 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-17T20:44:10.908 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr. 2019-10-17T20:44:10.923 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../049-ceph-mgr_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:10.966 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:11.480 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mon. 2019-10-17T20:44:11.495 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../050-ceph-mon_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:11.545 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:12.221 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph. 2019-10-17T20:44:12.237 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../052-ceph_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:12.279 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:12.442 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-osd. 2019-10-17T20:44:12.456 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../051-ceph-osd_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:12.500 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-osd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:12.548 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package runc. 2019-10-17T20:44:12.564 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../053-runc_1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2_amd64.deb ... 2019-10-17T20:44:12.606 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-17T20:44:13.193 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package containerd. 2019-10-17T20:44:13.206 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../054-containerd_1.2.6-0ubuntu1~18.04.2_amd64.deb ... 2019-10-17T20:44:13.259 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-17T20:44:14.355 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph. 2019-10-17T20:44:14.368 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../052-ceph_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:14.413 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:14.657 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package runc. 2019-10-17T20:44:14.670 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../053-runc_1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2_amd64.deb ... 2019-10-17T20:44:14.707 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-17T20:44:15.180 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package docker.io. 2019-10-17T20:44:15.193 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../055-docker.io_18.09.7-0ubuntu1~18.04.4_amd64.deb ... 2019-10-17T20:44:15.227 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package containerd. 2019-10-17T20:44:15.241 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../054-containerd_1.2.6-0ubuntu1~18.04.2_amd64.deb ... 2019-10-17T20:44:15.260 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-17T20:44:15.277 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-17T20:44:17.181 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package docker.io. 2019-10-17T20:44:17.195 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../055-docker.io_18.09.7-0ubuntu1~18.04.4_amd64.deb ... 2019-10-17T20:44:17.269 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-17T20:44:18.200 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-daemon. 2019-10-17T20:44:18.214 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../056-ceph-daemon_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:18.257 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-daemon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:18.604 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-fuse. 2019-10-17T20:44:18.618 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../057-ceph-fuse_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:18.662 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:19.032 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mds. 2019-10-17T20:44:19.048 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../058-ceph-mds_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:19.090 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mds (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:19.612 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-repoze.lru. 2019-10-17T20:44:19.625 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../059-python-repoze.lru_0.7-1_all.deb ... 2019-10-17T20:44:19.677 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-10-17T20:44:19.921 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjs-underscore. 2019-10-17T20:44:19.934 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../060-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-10-17T20:44:19.979 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-10-17T20:44:20.163 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-daemon. 2019-10-17T20:44:20.176 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../056-ceph-daemon_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:20.220 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-daemon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:20.240 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-10-17T20:44:20.254 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-10-17T20:44:20.298 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-17T20:44:20.534 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-fuse. 2019-10-17T20:44:20.547 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../057-ceph-fuse_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:20.590 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:20.618 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-routes. 2019-10-17T20:44:20.633 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../062-python-routes_2.4.1-1_all.deb ... 2019-10-17T20:44:20.676 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-routes (2.4.1-1) ... 2019-10-17T20:44:21.010 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mds. 2019-10-17T20:44:21.026 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../058-ceph-mds_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:21.029 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-10-17T20:44:21.044 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:21.076 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mds (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:21.087 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-dashboard (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:21.571 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-repoze.lru. 2019-10-17T20:44:21.584 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../059-python-repoze.lru_0.7-1_all.deb ... 2019-10-17T20:44:21.621 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-repoze.lru (0.7-1) ... 2019-10-17T20:44:21.874 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjs-underscore. 2019-10-17T20:44:21.887 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../060-libjs-underscore_1.8.3~dfsg-1_all.deb ... 2019-10-17T20:44:21.932 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjs-underscore (1.8.3~dfsg-1) ... 2019-10-17T20:44:21.943 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-10-17T20:44:21.956 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-cloud_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:22.026 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:22.210 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjs-sphinxdoc. 2019-10-17T20:44:22.226 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.6.7-1ubuntu1_all.deb ... 2019-10-17T20:44:22.268 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-17T20:44:22.337 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-decorator. 2019-10-17T20:44:22.354 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../065-python-decorator_4.1.2-1_all.deb ... 2019-10-17T20:44:22.395 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-10-17T20:44:22.604 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-routes. 2019-10-17T20:44:22.621 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../062-python-routes_2.4.1-1_all.deb ... 2019-10-17T20:44:22.663 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-routes (2.4.1-1) ... 2019-10-17T20:44:22.732 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-scipy. 2019-10-17T20:44:22.746 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../066-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-10-17T20:44:22.790 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-10-17T20:44:23.008 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2019-10-17T20:44:23.023 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:23.066 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-dashboard (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:23.923 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-sklearn-lib. 2019-10-17T20:44:23.941 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../067-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-10-17T20:44:23.953 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2019-10-17T20:44:23.962 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-cloud_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:23.982 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-10-17T20:44:24.006 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:24.334 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-decorator. 2019-10-17T20:44:24.348 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../065-python-decorator_4.1.2-1_all.deb ... 2019-10-17T20:44:24.392 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-decorator (4.1.2-1) ... 2019-10-17T20:44:24.468 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-joblib. 2019-10-17T20:44:24.482 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../068-python-joblib_0.11-1_all.deb ... 2019-10-17T20:44:24.526 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-joblib (0.11-1) ... 2019-10-17T20:44:24.704 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-scipy. 2019-10-17T20:44:24.719 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../066-python-scipy_0.19.1-2ubuntu1_amd64.deb ... 2019-10-17T20:44:24.762 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-scipy (0.19.1-2ubuntu1) ... 2019-10-17T20:44:24.904 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-sklearn. 2019-10-17T20:44:24.918 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../069-python-sklearn_0.19.1-3_all.deb ... 2019-10-17T20:44:24.962 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-10-17T20:44:25.485 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-10-17T20:44:25.499 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:25.569 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:25.945 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sklearn-lib. 2019-10-17T20:44:25.961 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../067-python-sklearn-lib_0.19.1-3_amd64.deb ... 2019-10-17T20:44:26.003 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sklearn-lib (0.19.1-3) ... 2019-10-17T20:44:26.263 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-monotonic. 2019-10-17T20:44:26.279 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../071-python-monotonic_1.1-2_all.deb ... 2019-10-17T20:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-monotonic (1.1-2) ... 2019-10-17T20:44:26.465 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-joblib. 2019-10-17T20:44:26.481 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../068-python-joblib_0.11-1_all.deb ... 2019-10-17T20:44:26.523 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-joblib (0.11-1) ... 2019-10-17T20:44:26.607 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-fasteners. 2019-10-17T20:44:26.623 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../072-python-fasteners_0.12.0-3_all.deb ... 2019-10-17T20:44:26.666 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-10-17T20:44:26.877 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sklearn. 2019-10-17T20:44:26.891 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../069-python-sklearn_0.19.1-3_all.deb ... 2019-10-17T20:44:26.936 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sklearn (0.19.1-3) ... 2019-10-17T20:44:26.986 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-httplib2. 2019-10-17T20:44:27.001 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../073-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-10-17T20:44:27.044 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-17T20:44:27.355 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pyasn1. 2019-10-17T20:44:27.369 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../074-python-pyasn1_0.4.2-3_all.deb ... 2019-10-17T20:44:27.422 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-10-17T20:44:27.470 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2019-10-17T20:44:27.484 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:27.527 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:27.751 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-10-17T20:44:27.767 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../075-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-10-17T20:44:27.809 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-10-17T20:44:28.154 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-rsa. 2019-10-17T20:44:28.169 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../076-python-rsa_3.4.2-1_all.deb ... 2019-10-17T20:44:28.212 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-10-17T20:44:28.231 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-monotonic. 2019-10-17T20:44:28.244 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../071-python-monotonic_1.1-2_all.deb ... 2019-10-17T20:44:28.289 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-monotonic (1.1-2) ... 2019-10-17T20:44:28.506 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-sqlalchemy. 2019-10-17T20:44:28.520 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../077-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-10-17T20:44:28.573 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:28.592 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-fasteners. 2019-10-17T20:44:28.607 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../072-python-fasteners_0.12.0-3_all.deb ... 2019-10-17T20:44:28.650 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-fasteners (0.12.0-3) ... 2019-10-17T20:44:28.937 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-httplib2. 2019-10-17T20:44:28.953 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../073-python-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2019-10-17T20:44:28.995 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-17T20:44:29.002 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-oauth2client. 2019-10-17T20:44:29.018 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../078-python-oauth2client_4.1.2-3_all.deb ... 2019-10-17T20:44:29.061 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-10-17T20:44:29.306 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pyasn1. 2019-10-17T20:44:29.320 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../074-python-pyasn1_0.4.2-3_all.deb ... 2019-10-17T20:44:29.365 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pyasn1 (0.4.2-3) ... 2019-10-17T20:44:29.470 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-websocket. 2019-10-17T20:44:29.487 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../079-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-10-17T20:44:29.556 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-10-17T20:44:29.711 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pyasn1-modules. 2019-10-17T20:44:29.725 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../075-python-pyasn1-modules_0.2.1-0.2_all.deb ... 2019-10-17T20:44:29.769 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pyasn1-modules (0.2.1-0.2) ... 2019-10-17T20:44:29.992 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-yaml. 2019-10-17T20:44:30.006 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../080-python-yaml_3.12-1build2_amd64.deb ... 2019-10-17T20:44:30.050 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-10-17T20:44:30.189 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-rsa. 2019-10-17T20:44:30.204 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../076-python-rsa_3.4.2-1_all.deb ... 2019-10-17T20:44:30.247 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-rsa (3.4.2-1) ... 2019-10-17T20:44:30.403 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-kubernetes. 2019-10-17T20:44:30.419 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../081-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-10-17T20:44:30.463 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-17T20:44:30.559 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sqlalchemy. 2019-10-17T20:44:30.573 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../077-python-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2019-10-17T20:44:30.623 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:30.898 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-10-17T20:44:30.912 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../082-ceph-mgr-k8sevents_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:30.956 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:31.080 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-oauth2client. 2019-10-17T20:44:31.095 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../078-python-oauth2client_4.1.2-3_all.deb ... 2019-10-17T20:44:31.138 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-oauth2client (4.1.2-3) ... 2019-10-17T20:44:31.251 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-10-17T20:44:31.264 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../083-ceph-mgr-rook_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:31.318 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-rook (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:31.483 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-websocket. 2019-10-17T20:44:31.499 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../079-python-websocket_0.44.0-0ubuntu2_all.deb ... 2019-10-17T20:44:31.542 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-websocket (0.44.0-0ubuntu2) ... 2019-10-17T20:44:31.637 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-10-17T20:44:31.651 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../084-ceph-mgr-ssh_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:31.695 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-mgr-ssh (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:31.938 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-yaml. 2019-10-17T20:44:31.953 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../080-python-yaml_3.12-1build2_amd64.deb ... 2019-10-17T20:44:31.996 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-yaml (3.12-1build2) ... 2019-10-17T20:44:32.048 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libonig4:amd64. 2019-10-17T20:44:32.062 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../085-libonig4_6.7.0-1_amd64.deb ... 2019-10-17T20:44:32.106 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-10-17T20:44:32.340 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-kubernetes. 2019-10-17T20:44:32.355 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../081-python-kubernetes_2.0.0-2ubuntu1_all.deb ... 2019-10-17T20:44:32.399 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-17T20:44:32.451 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libjq1:amd64. 2019-10-17T20:44:32.465 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../086-libjq1_1.5+dfsg-2_amd64.deb ... 2019-10-17T20:44:32.509 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-10-17T20:44:32.803 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package jq. 2019-10-17T20:44:32.817 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../087-jq_1.5+dfsg-2_amd64.deb ... 2019-10-17T20:44:32.845 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2019-10-17T20:44:32.860 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../082-ceph-mgr-k8sevents_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:32.861 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-10-17T20:44:32.903 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:33.131 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package socat. 2019-10-17T20:44:33.147 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../088-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-10-17T20:44:33.181 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-rook. 2019-10-17T20:44:33.189 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-10-17T20:44:33.197 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../083-ceph-mgr-rook_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:33.239 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-rook (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:33.542 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-mgr-ssh. 2019-10-17T20:44:33.556 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../084-ceph-mgr-ssh_15.0.0-6144-gf0009d4-1bionic_all.deb ... 2019-10-17T20:44:33.567 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-10-17T20:44:33.582 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../089-libxslt1.1_1.1.29-5ubuntu0.1_amd64.deb ... 2019-10-17T20:44:33.600 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-mgr-ssh (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:33.634 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-17T20:44:33.920 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package xmlstarlet. 2019-10-17T20:44:33.936 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2_amd64.deb ... 2019-10-17T20:44:33.978 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-10-17T20:44:33.979 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libonig4:amd64. 2019-10-17T20:44:33.994 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../085-libonig4_6.7.0-1_amd64.deb ... 2019-10-17T20:44:34.036 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2019-10-17T20:44:34.324 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ceph-test. 2019-10-17T20:44:34.338 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../091-ceph-test_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:34.382 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ceph-test (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:34.398 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libjq1:amd64. 2019-10-17T20:44:34.414 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../086-libjq1_1.5+dfsg-2_amd64.deb ... 2019-10-17T20:44:34.456 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2019-10-17T20:44:34.726 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package jq. 2019-10-17T20:44:34.740 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../087-jq_1.5+dfsg-2_amd64.deb ... 2019-10-17T20:44:34.784 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking jq (1.5+dfsg-2) ... 2019-10-17T20:44:35.045 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package socat. 2019-10-17T20:44:35.059 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../088-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2019-10-17T20:44:35.103 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2019-10-17T20:44:35.432 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libxslt1.1:amd64. 2019-10-17T20:44:35.445 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../089-libxslt1.1_1.1.29-5ubuntu0.1_amd64.deb ... 2019-10-17T20:44:35.482 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-17T20:44:35.726 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package xmlstarlet. 2019-10-17T20:44:35.742 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2_amd64.deb ... 2019-10-17T20:44:35.785 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2019-10-17T20:44:36.097 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ceph-test. 2019-10-17T20:44:36.114 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../091-ceph-test_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:36.147 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ceph-test (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:37.236 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package cgroupfs-mount. 2019-10-17T20:44:37.250 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../092-cgroupfs-mount_1.4_all.deb ... 2019-10-17T20:44:37.304 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking cgroupfs-mount (1.4) ... 2019-10-17T20:44:37.572 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package dns-root-data. 2019-10-17T20:44:37.588 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../093-dns-root-data_2018013001_all.deb ... 2019-10-17T20:44:37.630 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking dns-root-data (2018013001) ... 2019-10-17T20:44:37.958 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package dnsmasq-base. 2019-10-17T20:44:37.972 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../094-dnsmasq-base_2.79-1_amd64.deb ... 2019-10-17T20:44:38.016 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking dnsmasq-base (2.79-1) ... 2019-10-17T20:44:38.349 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package libcephfs-dev. 2019-10-17T20:44:38.363 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../095-libcephfs-dev_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:38.405 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking libcephfs-dev (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:38.750 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-attr. 2019-10-17T20:44:38.766 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../096-python-attr_17.4.0-2_all.deb ... 2019-10-17T20:44:38.808 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-attr (17.4.0-2) ... 2019-10-17T20:44:38.986 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package cgroupfs-mount. 2019-10-17T20:44:38.998 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../092-cgroupfs-mount_1.4_all.deb ... 2019-10-17T20:44:39.053 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking cgroupfs-mount (1.4) ... 2019-10-17T20:44:39.060 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-ceph. 2019-10-17T20:44:39.075 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../097-python-ceph_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:39.119 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:39.296 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package dns-root-data. 2019-10-17T20:44:39.310 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../093-dns-root-data_2018013001_all.deb ... 2019-10-17T20:44:39.354 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking dns-root-data (2018013001) ... 2019-10-17T20:44:39.421 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-funcsigs. 2019-10-17T20:44:39.438 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../098-python-funcsigs_1.0.2-4_all.deb ... 2019-10-17T20:44:39.479 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-10-17T20:44:39.624 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package dnsmasq-base. 2019-10-17T20:44:39.638 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../094-dnsmasq-base_2.79-1_amd64.deb ... 2019-10-17T20:44:39.674 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking dnsmasq-base (2.79-1) ... 2019-10-17T20:44:39.841 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-webencodings. 2019-10-17T20:44:39.856 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../099-python-webencodings_0.5-2_all.deb ... 2019-10-17T20:44:39.899 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-webencodings (0.5-2) ... 2019-10-17T20:44:39.994 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package libcephfs-dev. 2019-10-17T20:44:40.010 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../095-libcephfs-dev_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:40.227 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-html5lib. 2019-10-17T20:44:40.241 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../100-python-html5lib_0.999999999-1_all.deb ... 2019-10-17T20:44:40.285 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-10-17T20:44:40.436 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking libcephfs-dev (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:40.632 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-lxml:amd64. 2019-10-17T20:44:40.645 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../101-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-10-17T20:44:40.688 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-17T20:44:40.747 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-attr. 2019-10-17T20:44:40.761 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../096-python-attr_17.4.0-2_all.deb ... 2019-10-17T20:44:40.805 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-attr (17.4.0-2) ... 2019-10-17T20:44:41.033 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-ceph. 2019-10-17T20:44:41.050 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../097-python-ceph_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:41.083 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:41.166 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-openid. 2019-10-17T20:44:41.181 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../102-python-openid_2.2.5-6_all.deb ... 2019-10-17T20:44:41.224 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-openid (2.2.5-6) ... 2019-10-17T20:44:41.352 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-funcsigs. 2019-10-17T20:44:41.367 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../098-python-funcsigs_1.0.2-4_all.deb ... 2019-10-17T20:44:41.410 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2019-10-17T20:44:41.561 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pastescript. 2019-10-17T20:44:41.577 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../103-python-pastescript_2.0.2-2_all.deb ... 2019-10-17T20:44:41.619 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-10-17T20:44:41.713 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-webencodings. 2019-10-17T20:44:41.728 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../099-python-webencodings_0.5-2_all.deb ... 2019-10-17T20:44:41.772 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-webencodings (0.5-2) ... 2019-10-17T20:44:41.923 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pluggy. 2019-10-17T20:44:41.939 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../104-python-pluggy_0.6.0-1_all.deb ... 2019-10-17T20:44:41.981 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-10-17T20:44:42.050 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-html5lib. 2019-10-17T20:44:42.066 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../100-python-html5lib_0.999999999-1_all.deb ... 2019-10-17T20:44:42.099 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-html5lib (0.999999999-1) ... 2019-10-17T20:44:42.309 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-py. 2019-10-17T20:44:42.324 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../105-python-py_1.5.2-1_all.deb ... 2019-10-17T20:44:42.367 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-py (1.5.2-1) ... 2019-10-17T20:44:42.419 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-lxml:amd64. 2019-10-17T20:44:42.436 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../101-python-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2019-10-17T20:44:42.469 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-17T20:44:42.695 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pyinotify. 2019-10-17T20:44:42.710 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../106-python-pyinotify_0.9.6-1_all.deb ... 2019-10-17T20:44:42.754 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-10-17T20:44:42.858 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-openid. 2019-10-17T20:44:42.873 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../102-python-openid_2.2.5-6_all.deb ... 2019-10-17T20:44:42.907 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-openid (2.2.5-6) ... 2019-10-17T20:44:43.065 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-pytest. 2019-10-17T20:44:43.082 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../107-python-pytest_3.3.2-2_all.deb ... 2019-10-17T20:44:43.123 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-10-17T20:44:43.201 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pastescript. 2019-10-17T20:44:43.216 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../103-python-pastescript_2.0.2-2_all.deb ... 2019-10-17T20:44:43.251 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pastescript (2.0.2-2) ... 2019-10-17T20:44:43.485 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-scgi. 2019-10-17T20:44:43.498 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../108-python-scgi_1.13-1.1build1_amd64.deb ... 2019-10-17T20:44:43.543 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-10-17T20:44:43.589 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pluggy. 2019-10-17T20:44:43.604 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../104-python-pluggy_0.6.0-1_all.deb ... 2019-10-17T20:44:43.647 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pluggy (0.6.0-1) ... 2019-10-17T20:44:43.796 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-10-17T20:44:43.810 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../109-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-10-17T20:44:43.854 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:43.950 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-py. 2019-10-17T20:44:43.967 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../105-python-py_1.5.2-1_all.deb ... 2019-10-17T20:44:44.008 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-py (1.5.2-1) ... 2019-10-17T20:44:44.207 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package radosgw. 2019-10-17T20:44:44.222 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../110-radosgw_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:44.273 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking radosgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:44.336 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pyinotify. 2019-10-17T20:44:44.351 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../106-python-pyinotify_0.9.6-1_all.deb ... 2019-10-17T20:44:44.394 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pyinotify (0.9.6-1) ... 2019-10-17T20:44:44.714 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-pytest. 2019-10-17T20:44:44.729 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../107-python-pytest_3.3.2-2_all.deb ... 2019-10-17T20:44:44.774 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-pytest (3.3.2-2) ... 2019-10-17T20:44:45.101 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-scgi. 2019-10-17T20:44:45.117 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../108-python-scgi_1.13-1.1build1_amd64.deb ... 2019-10-17T20:44:45.167 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-scgi (1.13-1.1build1) ... 2019-10-17T20:44:45.326 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package rbd-fuse. 2019-10-17T20:44:45.341 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../111-rbd-fuse_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:45.384 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking rbd-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:45.437 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package python-sqlalchemy-ext. 2019-10-17T20:44:45.454 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../109-python-sqlalchemy-ext_1.1.11+ds1-1ubuntu1_amd64.deb ... 2019-10-17T20:44:45.495 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:45.754 INFO:teuthology.orchestra.run.smithi092.stdout:Selecting previously unselected package ubuntu-fan. 2019-10-17T20:44:45.769 INFO:teuthology.orchestra.run.smithi092.stdout:Preparing to unpack .../112-ubuntu-fan_0.12.10_all.deb ... 2019-10-17T20:44:45.821 INFO:teuthology.orchestra.run.smithi092.stdout:Unpacking ubuntu-fan (0.12.10) ... 2019-10-17T20:44:45.849 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package radosgw. 2019-10-17T20:44:45.862 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../110-radosgw_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:45.907 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking radosgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:46.125 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-10-17T20:44:46.368 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-10-17T20:44:46.595 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-10-17T20:44:46.602 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package rbd-fuse. 2019-10-17T20:44:46.618 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../111-rbd-fuse_15.0.0-6144-gf0009d4-1bionic_amd64.deb ... 2019-10-17T20:44:46.660 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking rbd-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:46.796 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:47.038 INFO:teuthology.orchestra.run.smithi204.stdout:Selecting previously unselected package ubuntu-fan. 2019-10-17T20:44:47.052 INFO:teuthology.orchestra.run.smithi204.stdout:Preparing to unpack .../112-ubuntu-fan_0.12.10_all.deb ... 2019-10-17T20:44:47.097 INFO:teuthology.orchestra.run.smithi204.stdout:Unpacking ubuntu-fan (0.12.10) ... 2019-10-17T20:44:47.265 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-openid (2.2.5-6) ... 2019-10-17T20:44:47.394 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-dnspython (1.15.0-1) ... 2019-10-17T20:44:47.491 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-10-17T20:44:47.628 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-simplejson (3.13.2-1) ... 2019-10-17T20:44:47.692 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-17T20:44:47.846 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-repoze.lru (0.7-1) ... 2019-10-17T20:44:47.911 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-yaml (3.12-1build2) ... 2019-10-17T20:44:48.048 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:48.162 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-10-17T20:44:48.355 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-chardet (3.0.4-1) ... 2019-10-17T20:44:48.533 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-openid (2.2.5-6) ... 2019-10-17T20:44:48.632 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-attr (17.4.0-2) ... 2019-10-17T20:44:48.759 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-prettytable (0.7.2-3) ... 2019-10-17T20:44:48.839 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-10-17T20:44:48.961 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-urllib3 (1.22-1ubuntu0.18.04.1) ... 2019-10-17T20:44:48.965 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-17T20:44:49.108 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-17T20:44:49.200 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-10-17T20:44:49.204 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-yaml (3.12-1build2) ... 2019-10-17T20:44:49.426 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-simplegeneric (0.8.1-1) ... 2019-10-17T20:44:49.453 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-10-17T20:44:49.577 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-10-17T20:44:49.624 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-chardet (3.0.4-1) ... 2019-10-17T20:44:49.820 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-py (1.5.2-1) ... 2019-10-17T20:44:49.900 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-attr (17.4.0-2) ... 2019-10-17T20:44:50.047 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-monotonic (1.1-2) ... 2019-10-17T20:44:50.101 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjs-underscore (1.8.3~dfsg-1) ... 2019-10-17T20:44:50.227 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up runc (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2) ... 2019-10-17T20:44:50.240 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-joblib (0.11-1) ... 2019-10-17T20:44:50.368 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-17T20:44:50.469 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-openssl (17.5.0-1ubuntu1) ... 2019-10-17T20:44:50.537 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-10-17T20:44:50.666 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-10-17T20:44:50.704 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2019-10-17T20:44:50.830 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pyasn1 (0.4.2-3) ... 2019-10-17T20:44:50.860 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-10-17T20:44:51.049 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-py (1.5.2-1) ... 2019-10-17T20:44:51.055 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up dns-root-data (2018013001) ... 2019-10-17T20:44:51.190 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-17T20:44:51.275 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-monotonic (1.1-2) ... 2019-10-17T20:44:51.320 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-17T20:44:51.446 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-10-17T20:44:51.459 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-joblib (0.11-1) ... 2019-10-17T20:44:51.697 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-ceph-argparse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:51.728 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2019-10-17T20:44:51.837 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-waitress (1.0.1-1) ... 2019-10-17T20:44:51.862 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pluggy (0.6.0-1) ... 2019-10-17T20:44:51.999 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-10-17T20:44:52.055 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-certifi (2018.1.18-2) ... 2019-10-17T20:44:52.212 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up cgroupfs-mount (1.4) ... 2019-10-17T20:44:52.246 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up dns-root-data (2018013001) ... 2019-10-17T20:44:52.374 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up formencode-i18n (1.3.0-0ubuntu5) ... 2019-10-17T20:44:52.502 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjs-sphinxdoc (1.6.7-1ubuntu1) ... 2019-10-17T20:44:52.630 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-webob (1:1.7.3-2fakesync1) ... 2019-10-17T20:44:52.662 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-17T20:44:52.868 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-ceph-argparse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:52.872 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service -> /lib/systemd/system/containerd.service. 2019-10-17T20:44:53.004 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-waitress (1.0.1-1) ... 2019-10-17T20:44:53.176 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-10-17T20:44:53.183 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2019-10-17T20:44:53.385 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up cgroupfs-mount (1.4) ... 2019-10-17T20:44:53.489 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-10-17T20:44:53.693 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-10-17T20:44:53.871 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up containerd (1.2.6-0ubuntu1~18.04.2) ... 2019-10-17T20:44:53.903 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-10-17T20:44:54.069 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service -> /lib/systemd/system/containerd.service. 2019-10-17T20:44:54.101 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-17T20:44:54.355 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up bridge-utils (1.5-15ubuntu1) ... 2019-10-17T20:44:54.407 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pyasn1-modules (0.2.1-0.2) ... 2019-10-17T20:44:54.627 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-10-17T20:44:54.717 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-fasteners (0.12.0-3) ... 2019-10-17T20:44:54.817 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-10-17T20:44:54.912 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-jwt (1.5.3+ds1-1) ... 2019-10-17T20:44:55.019 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-10-17T20:44:55.111 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-10-17T20:44:55.128 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-markupsafe (1.0-1build1) ... 2019-10-17T20:44:55.273 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-17T20:44:55.340 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-werkzeug (0.14.1+dfsg1-1) ... 2019-10-17T20:44:55.402 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-17T20:44:55.599 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-10-17T20:44:55.601 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up bridge-utils (1.5-15ubuntu1) ... 2019-10-17T20:44:55.725 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:55.860 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-17T20:44:55.950 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pyinotify (0.9.6-1) ... 2019-10-17T20:44:55.995 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-10-17T20:44:56.147 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-scgi (1.13-1.1build1) ... 2019-10-17T20:44:56.230 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-10-17T20:44:56.341 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-websocket (0.44.0-0ubuntu2) ... 2019-10-17T20:44:56.408 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/python2-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2019-10-17T20:44:56.459 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-10-17T20:44:56.555 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.1) ... 2019-10-17T20:44:56.579 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-17T20:44:56.692 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2019-10-17T20:44:56.748 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-17T20:44:56.886 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2019-10-17T20:44:56.961 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-17T20:44:57.023 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:44:57.137 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-17T20:44:57.167 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:57.272 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-bs4 (4.6.0-1) ... 2019-10-17T20:44:57.287 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-10-17T20:44:57.467 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up dnsmasq-base (2.79-1) ... 2019-10-17T20:44:57.499 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-mako (1.0.7+ds1-1) ... 2019-10-17T20:44:57.700 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sklearn-lib (0.19.1-3) ... 2019-10-17T20:44:57.801 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2019-10-17T20:44:57.947 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-17T20:44:58.135 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-10-17T20:44:58.140 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2019-10-17T20:44:58.348 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-17T20:44:58.361 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sqlalchemy-ext (1.1.11+ds1-1ubuntu1) ... 2019-10-17T20:44:58.482 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-funcsigs (1.0.2-4) ... 2019-10-17T20:44:58.677 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up dnsmasq-base (2.79-1) ... 2019-10-17T20:44:59.297 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-requests (2.18.4-2ubuntu0.1) ... 2019-10-17T20:44:59.509 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2019-10-17T20:45:02.418 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-10-17T20:45:02.594 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-decorator (4.1.2-1) ... 2019-10-17T20:45:02.786 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liboath0 (2.6.1-1) ... 2019-10-17T20:45:02.955 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-10-17T20:45:03.148 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-17T20:45:03.248 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-10-17T20:45:03.350 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.1) ... 2019-10-17T20:45:03.425 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-tempita (0.5.2-2) ... 2019-10-17T20:45:03.430 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-uritemplate (0.6-1ubuntu1) ... 2019-10-17T20:45:03.623 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-decorator (4.1.2-1) ... 2019-10-17T20:45:03.627 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-17T20:45:03.752 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up pigz (2.4-1) ... 2019-10-17T20:45:03.815 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liboath0 (2.6.1-1) ... 2019-10-17T20:45:03.878 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-webencodings (0.5-2) ... 2019-10-17T20:45:03.967 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-singledispatch (3.4.0.3-2) ... 2019-10-17T20:45:04.072 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-10-17T20:45:04.135 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.1) ... 2019-10-17T20:45:04.244 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2019-10-17T20:45:04.331 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-10-17T20:45:04.370 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.1) ... 2019-10-17T20:45:04.454 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-tempita (0.5.2-2) ... 2019-10-17T20:45:04.499 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-logutils (0.3.3-5) ... 2019-10-17T20:45:04.655 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2019-10-17T20:45:04.683 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-routes (2.4.1-1) ... 2019-10-17T20:45:04.781 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up pigz (2.4-1) ... 2019-10-17T20:45:04.884 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-10-17T20:45:04.907 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-webencodings (0.5-2) ... 2019-10-17T20:45:05.092 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-cherrypy3 (8.9.1-2) ... 2019-10-17T20:45:05.119 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up jq (1.5+dfsg-2) ... 2019-10-17T20:45:05.245 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-17T20:45:05.335 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-bcrypt (3.1.4-2) ... 2019-10-17T20:45:05.503 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-logutils (0.3.3-5) ... 2019-10-17T20:45:05.671 INFO:teuthology.orchestra.run.smithi092.stdout:Adding group `docker' (GID 122) ... 2019-10-17T20:45:05.704 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-routes (2.4.1-1) ... 2019-10-17T20:45:05.760 INFO:teuthology.orchestra.run.smithi092.stdout:Done. 2019-10-17T20:45:05.893 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/sockets.target.wants/docker.socket -> /lib/systemd/system/docker.socket. 2019-10-17T20:45:05.905 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-formencode (1.3.0-0ubuntu5) ... 2019-10-17T20:45:06.139 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up jq (1.5+dfsg-2) ... 2019-10-17T20:45:06.265 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up docker.io (18.09.7-0ubuntu1~18.04.4) ... 2019-10-17T20:45:06.648 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group `docker' (GID 122) ... 2019-10-17T20:45:06.730 INFO:teuthology.orchestra.run.smithi204.stdout:Done. 2019-10-17T20:45:06.865 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/sockets.target.wants/docker.socket -> /lib/systemd/system/docker.socket. 2019-10-17T20:45:09.082 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-10-17T20:45:10.011 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-scipy (0.19.1-2ubuntu1) ... 2019-10-17T20:45:10.061 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-rsa (3.4.2-1) ... 2019-10-17T20:45:10.272 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-17T20:45:10.555 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pytest (3.3.2-2) ... 2019-10-17T20:45:10.807 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-10-17T20:45:10.991 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-10-17T20:45:11.007 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rsa (3.4.2-1) ... 2019-10-17T20:45:11.109 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-10-17T20:45:11.216 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-paste (2.0.3+dfsg-4ubuntu1) ... 2019-10-17T20:45:11.235 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librados2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:11.446 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcephfs2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:11.493 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pytest (3.3.2-2) ... 2019-10-17T20:45:11.572 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-10-17T20:45:11.736 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-oauth2client (4.1.2-3) ... 2019-10-17T20:45:11.945 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2019-10-17T20:45:12.071 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up xmlstarlet (1.6.1-2) ... 2019-10-17T20:45:12.197 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librados2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:12.225 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ubuntu-fan (0.12.10) ... 2019-10-17T20:45:12.409 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:12.518 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-sklearn (0.19.1-3) ... 2019-10-17T20:45:12.840 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service -> /lib/systemd/system/ubuntu-fan.service. 2019-10-17T20:45:13.180 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ubuntu-fan (0.12.10) ... 2019-10-17T20:45:13.272 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libcephfs-dev (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:13.389 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-daemon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:13.490 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-17T20:45:13.730 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service -> /lib/systemd/system/ubuntu-fan.service. 2019-10-17T20:45:14.018 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-10-17T20:45:14.177 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libcephfs-dev (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.286 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-daemon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.319 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-rados (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.403 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-kubernetes (2.0.0-2ubuntu1) ... 2019-10-17T20:45:14.446 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-cephfs (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.638 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up libradosstriper1 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.764 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librgw2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.865 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-rgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:14.948 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-html5lib (0.999999999-1) ... 2019-10-17T20:45:14.966 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-10-17T20:45:15.151 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.182 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rados (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.308 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-cephfs (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.371 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-17T20:45:15.371 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-17T20:45:15.485 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up libradosstriper1 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.586 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librgw2 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.660 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up librbd1 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.687 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:15.795 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-10-17T20:45:15.812 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastedeploy (1.5.2-4) ... 2019-10-17T20:45:16.006 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:16.021 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-rbd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:16.147 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:16.223 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-17T20:45:16.223 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2019-10-17T20:45:16.507 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up librbd1 (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:16.616 INFO:teuthology.orchestra.run.smithi092.stdout:Adding group ceph....done 2019-10-17T20:45:16.633 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pastescript (2.0.2-2) ... 2019-10-17T20:45:16.843 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-rbd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:16.944 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-common (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:17.018 INFO:teuthology.orchestra.run.smithi092.stdout:Adding system user ceph....done 2019-10-17T20:45:17.194 INFO:teuthology.orchestra.run.smithi092.stdout:Setting system user ceph properties....done 2019-10-17T20:45:17.356 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-10-17T20:45:17.453 INFO:teuthology.orchestra.run.smithi204.stdout:Adding group ceph....done 2019-10-17T20:45:17.584 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-10-17T20:45:17.814 INFO:teuthology.orchestra.run.smithi204.stdout:Adding system user ceph....done 2019-10-17T20:45:17.883 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up radosgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:17.973 INFO:teuthology.orchestra.run.smithi204.stdout:Setting system user ceph properties....done 2019-10-17T20:45:18.126 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2019-10-17T20:45:18.127 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-17T20:45:18.128 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-17T20:45:18.345 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2019-10-17T20:45:18.509 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up rbd-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:18.627 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:18.638 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up radosgw (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:18.754 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-10-17T20:45:18.894 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-17T20:45:18.894 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2019-10-17T20:45:18.964 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-test (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:19.114 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-base (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:19.281 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up rbd-fuse (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:19.412 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:19.444 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-10-17T20:45:19.533 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-webtest (2.0.28-1ubuntu1) ... 2019-10-17T20:45:19.741 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up python-pecan (1.2.1-2) ... 2019-10-17T20:45:19.743 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-test (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:19.814 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-10-17T20:45:19.844 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-base (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:19.845 INFO:teuthology.orchestra.run.smithi092.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-10-17T20:45:20.035 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-osd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:20.155 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2019-10-17T20:45:20.454 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up python-pecan (1.2.1-2) ... 2019-10-17T20:45:20.544 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/python2-pecan to provide /usr/bin/pecan (pecan) in auto mode 2019-10-17T20:45:20.576 INFO:teuthology.orchestra.run.smithi204.stdout:update-alternatives: using /usr/bin/python2-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2019-10-17T20:45:20.625 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-17T20:45:20.625 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-17T20:45:20.757 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-osd (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:20.888 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mds (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:21.102 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-17T20:45:21.103 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-17T20:45:21.316 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-17T20:45:21.316 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2019-10-17T20:45:21.372 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:21.593 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mds (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:21.606 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-17T20:45:21.606 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-17T20:45:21.799 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-17T20:45:21.799 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2019-10-17T20:45:21.882 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.077 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mon (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.088 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-17T20:45:22.088 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-17T20:45:22.274 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-17T20:45:22.274 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2019-10-17T20:45:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.494 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.554 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.644 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-rook (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.773 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-17T20:45:22.773 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2019-10-17T20:45:22.789 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:22.914 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-k8sevents (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.030 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.032 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-ssh (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.150 INFO:teuthology.orchestra.run.smithi092.stdout:Setting up ceph-mgr-dashboard (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.181 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.268 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-17T20:45:23.324 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-rook (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.351 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-17T20:45:23.467 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.483 INFO:teuthology.orchestra.run.smithi092.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-17T20:45:23.593 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-k8sevents (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.694 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-ssh (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.812 INFO:teuthology.orchestra.run.smithi204.stdout:Setting up ceph-mgr-dashboard (15.0.0-6144-gf0009d4-1bionic) ... 2019-10-17T20:45:23.938 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2019-10-17T20:45:24.005 INFO:teuthology.orchestra.run.smithi092.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-17T20:45:24.008 DEBUG:teuthology.parallel:result is None 2019-10-17T20:45:24.038 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2019-10-17T20:45:24.173 INFO:teuthology.orchestra.run.smithi204.stdout:Processing triggers for systemd (237-3ubuntu10.25) ... 2019-10-17T20:45:24.676 INFO:teuthology.orchestra.run.smithi204.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2019-10-17T20:45:24.679 DEBUG:teuthology.parallel:result is None 2019-10-17T20:45:24.679 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:24.679 INFO:teuthology.orchestra.run.smithi092:> sudo lsb_release -is 2019-10-17T20:45:24.734 INFO:teuthology.orchestra.run.smithi092.stdout:Ubuntu 2019-10-17T20:45:24.739 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-17T20:45:24.739 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 2019-10-17T20:45:24.905 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:24.906 INFO:teuthology.orchestra.run.smithi092:> dpkg-query -W -f '${Version}' ceph 2019-10-17T20:45:24.936 INFO:teuthology.orchestra.run.smithi092.stdout:15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:45:24.937 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:45:24.937 INFO:teuthology.task.install:The correct ceph version 15.0.0-6144-gf0009d4-1bionic is installed. 2019-10-17T20:45:24.937 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:24.937 INFO:teuthology.orchestra.run.smithi204:> sudo lsb_release -is 2019-10-17T20:45:24.989 INFO:teuthology.orchestra.run.smithi204.stdout:Ubuntu 2019-10-17T20:45:24.994 DEBUG:teuthology.misc:System to be installed: Ubuntu 2019-10-17T20:45:24.994 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f0009d4765d23f6e6a0d55d83c1e9f63c45550d5 2019-10-17T20:45:25.147 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.147 INFO:teuthology.orchestra.run.smithi204:> dpkg-query -W -f '${Version}' ceph 2019-10-17T20:45:25.172 INFO:teuthology.orchestra.run.smithi204.stdout:15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:45:25.172 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-6144-gf0009d4-1bionic 2019-10-17T20:45:25.172 INFO:teuthology.task.install:The correct ceph version 15.0.0-6144-gf0009d4-1bionic is installed. 2019-10-17T20:45:25.173 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-10-17T20:45:25.173 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.173 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-17T20:45:25.191 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.191 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-10-17T20:45:25.239 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-10-17T20:45:25.240 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.240 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-17T20:45:25.259 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.259 INFO:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-17T20:45:25.344 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.344 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-10-17T20:45:25.367 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.367 INFO:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-10-17T20:45:25.452 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-10-17T20:45:25.452 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.452 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-17T20:45:25.539 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.540 INFO:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-17T20:45:25.626 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.626 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-10-17T20:45:25.692 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.692 INFO:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-10-17T20:45:25.776 INFO:teuthology.run_tasks:Running task ceph... 2019-10-17T20:45:25.972 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-10-17T20:45:25.972 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.972 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2019-10-17T20:45:25.976 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:25.976 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2019-10-17T20:45:25.992 INFO:tasks.ceph:Disabling ceph logrotate... 2019-10-17T20:45:25.993 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:25.993 INFO:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-17T20:45:26.060 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.060 INFO:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2019-10-17T20:45:26.076 INFO:tasks.ceph:Creating extra log directories... 2019-10-17T20:45:26.076 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.076 INFO:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-17T20:45:26.144 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.144 INFO:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-10-17T20:45:26.160 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-10-17T20:45:26.161 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'ms inject socket failures': 5000, 'ms type': 'async', 'mon client directed command retry': 5}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd initial require min compat client': 'luminous', 'mon min osdmap epochs': 50, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'osd op queue cut off': 'debug_random', 'debug ms': 1, 'debug journal': 20, 'osd_discard_disconnected_ops': False, 'debug osd': 25, 'osd objectstore': 'filestore', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd op queue': 'debug_random', 'osd sloppy crc': True, 'debug filestore': 20, 'osd shutdown pgref assert': True}}} 2019-10-17T20:45:26.162 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'rados': {'op_weights': {'read': 45, 'write': 45, 'delete': 10}, 'clients': ['client.0'], 'objects': 500, 'ops': 4000}}], 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.4309', 'owner': 'scheduled_sjust@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'targets': {'smithi092.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvZvqdwAiqPH7Tv1SjnaVZbTB7C3O9xmcp8DXENfY2xFXll6gLqfr1/nmtdlJqRuSM68CE+wiFlpv/97UuOtY5a99TYfOzNqZL8qJxpfNs9wH12XDF2Jjyd9rLVwrH9pcARm4icefXOTSlntJt0YdOn/IJbjeeUulmAxLtMYzM/t4BD0TPrAi061MNPZamx99gWEZwZkN3FFTWHGNR+hKaZUqwSoFKGVxuSncL/Y/FJ27itleZRQTHTdPDbjeLe6e6yvgf/pLyAu/AvnIqrSyHBY+khT/2mYNqwfXY5mwZIBQRRGOI1RgRDxWuX7mp3oI5aCSE6BMFjw/NA5Dbv1RN', 'smithi204.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxcgkWcgpyXJpHPdcMl8MDrjKlUgM3Us9wefFAGIALtC4e2S4bWyY/L9DXlacl7CJCIIhy6oB7yzN/KZ35HJskpcwp6GAWk0PMRHnhOTC5xioN5fskgms3DKh5UXWqb9Ulyn4UWNRkKv20J5Lr43Jfo0qZ9QlSLRVf+IrVLturRG49z2bZYvi2yapEuI28/HezE9qpx5Hd1L5qWQ4Fmjpz3XVjrjMgDduhjCEbyuQj5xGp8xFREZB1fYTfJ6JocCgqCfxH58cWTEB+c2ZFxV5HFrN9/VWLTtWlVaS6N4qNNmy1uQh/qUA94Jqb02P8Svi2/7YwIsYBhixu6JCBvGu9'}, 'job_id': '4418197', 'tube': 'smithi', 'suite_branch': 'wip-sjust-testing2', 'priority': 90, 'os_version': '18.04', 'first_in_suite': False, 'branch': 'wip-sjust-testing2', 'suite': 'rados', 'email': None, 'archive_path': '/home/teuthworker/archive/sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi/4418197', 'description': 'rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/readwrite.yaml}', 'nuke-on-error': True, 'repo': 'https://github.com/ceph/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5', 'name': 'sjust-2019-10-17_19:24:08-rados-wip-sjust-testing2-distro-basic-smithi', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5', 'os_type': 'ubuntu', 'overrides': {'ceph': {'conf': {'global': {'ms inject socket failures': 5000, 'ms type': 'async', 'mon client directed command retry': 5}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd initial require min compat client': 'luminous', 'mon min osdmap epochs': 50, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'osd op queue cut off': 'debug_random', 'debug ms': 1, 'debug journal': 20, 'osd_discard_disconnected_ops': False, 'debug osd': 25, 'osd objectstore': 'filestore', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd op queue': 'debug_random', 'osd sloppy crc': True, 'debug filestore': 20, 'osd shutdown pgref assert': True}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'crush_tunables': 'optimal', 'fs': 'xfs', 'sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5'}, 'ceph-deploy': {'fs': 'xfs', 'filestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd sloppy crc': True, 'osd objectstore': 'filestore'}}}, 'workunit': {'sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5', 'branch': 'wip-sjust-testing2'}, 'install': {'ceph': {'sha1': 'f0009d4765d23f6e6a0d55d83c1e9f63c45550d5'}}, 'admin_socket': {'branch': 'wip-sjust-testing2'}}} 2019-10-17T20:45:26.162 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.162 INFO:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-17T20:45:26.232 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.232 INFO:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-10-17T20:45:26.240 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.240 INFO:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2019-10-17T20:45:26.313 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.313 INFO:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2019-10-17T20:45:26.403 DEBUG:teuthology.orchestra.remote:smithi092:/scratch_devs is 72B 2019-10-17T20:45:26.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-17T20:45:26.417 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.417 INFO:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-17T20:45:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-10-17T20:45:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.425 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 433 Links: 1 2019-10-17T20:45:26.426 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.426 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2019-10-17 20:42:42.711207384 +0000 2019-10-17T20:45:26.426 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2019-10-17 20:42:42.595207386 +0000 2019-10-17T20:45:26.426 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2019-10-17 20:42:42.595207386 +0000 2019-10-17T20:45:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2019-10-17T20:45:26.434 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2019-10-17T20:45:26.435 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2019-10-17T20:45:26.435 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000347571 s, 1.5 MB/s 2019-10-17T20:45:26.438 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.439 INFO:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-17T20:45:26.510 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-10-17T20:45:26.510 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.511 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 439 Links: 1 2019-10-17T20:45:26.511 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.511 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2019-10-17 20:42:42.955207382 +0000 2019-10-17T20:45:26.511 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2019-10-17 20:42:42.831207383 +0000 2019-10-17T20:45:26.511 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2019-10-17 20:42:42.831207383 +0000 2019-10-17T20:45:26.511 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2019-10-17T20:45:26.520 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2019-10-17T20:45:26.520 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2019-10-17T20:45:26.520 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.00037816 s, 1.4 MB/s 2019-10-17T20:45:26.525 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.525 INFO:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-17T20:45:26.596 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-10-17T20:45:26.597 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.597 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 445 Links: 1 2019-10-17T20:45:26.597 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.597 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2019-10-17 20:42:43.219207379 +0000 2019-10-17T20:45:26.597 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2019-10-17 20:42:43.103207380 +0000 2019-10-17T20:45:26.598 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2019-10-17 20:42:43.103207380 +0000 2019-10-17T20:45:26.598 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2019-10-17T20:45:26.606 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2019-10-17T20:45:26.606 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2019-10-17T20:45:26.606 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000359672 s, 1.4 MB/s 2019-10-17T20:45:26.610 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:26.610 INFO:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-17T20:45:26.681 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-10-17T20:45:26.681 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.681 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 451 Links: 1 2019-10-17T20:45:26.682 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.682 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2019-10-17 20:44:52.466173290 +0000 2019-10-17T20:45:26.682 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2019-10-17 20:42:43.379207377 +0000 2019-10-17T20:45:26.682 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2019-10-17 20:42:43.379207377 +0000 2019-10-17T20:45:26.682 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2019-10-17T20:45:26.690 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2019-10-17T20:45:26.690 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2019-10-17T20:45:26.691 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000340287 s, 1.5 MB/s 2019-10-17T20:45:26.695 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-17T20:45:26.695 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-17T20:45:26.695 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-10-17T20:45:26.696 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:26.705 DEBUG:teuthology.orchestra.remote:smithi204:/scratch_devs is 72B 2019-10-17T20:45:26.717 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-17T20:45:26.718 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.718 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-10-17T20:45:26.725 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2019-10-17T20:45:26.725 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.726 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 452 Links: 1 2019-10-17T20:45:26.726 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.726 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-17 20:42:42.842772398 +0000 2019-10-17T20:45:26.726 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-17 20:42:42.730828401 +0000 2019-10-17T20:45:26.726 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-17 20:42:42.730828401 +0000 2019-10-17T20:45:26.726 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-17T20:45:26.735 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-17T20:45:26.735 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-17T20:45:26.735 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000417748 s, 1.2 MB/s 2019-10-17T20:45:26.740 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.740 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-10-17T20:45:26.811 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2019-10-17T20:45:26.811 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.812 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 458 Links: 1 2019-10-17T20:45:26.812 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.812 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-17 20:42:43.054666393 +0000 2019-10-17T20:45:26.812 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-17 20:42:42.978704395 +0000 2019-10-17T20:45:26.812 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-17 20:42:42.978704395 +0000 2019-10-17T20:45:26.813 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-17T20:45:26.821 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-17T20:45:26.821 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-17T20:45:26.821 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000341107 s, 1.5 MB/s 2019-10-17T20:45:26.825 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.825 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-10-17T20:45:26.897 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2019-10-17T20:45:26.897 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.897 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 464 Links: 1 2019-10-17T20:45:26.898 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.898 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-17 20:42:43.178604390 +0000 2019-10-17T20:45:26.898 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-17 20:42:43.174606391 +0000 2019-10-17T20:45:26.898 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-17 20:42:43.174606391 +0000 2019-10-17T20:45:26.898 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-17T20:45:26.906 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-17T20:45:26.907 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-17T20:45:26.907 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000396213 s, 1.3 MB/s 2019-10-17T20:45:26.911 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:26.911 INFO:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-10-17T20:45:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2019-10-17T20:45:26.983 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-10-17T20:45:26.983 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 6h/6d Inode: 470 Links: 1 2019-10-17T20:45:26.984 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-10-17T20:45:26.984 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2019-10-17 20:42:43.394496386 +0000 2019-10-17T20:45:26.984 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2019-10-17 20:42:43.390498385 +0000 2019-10-17T20:45:26.984 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2019-10-17 20:42:43.390498385 +0000 2019-10-17T20:45:26.984 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: - 2019-10-17T20:45:26.993 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2019-10-17T20:45:26.993 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2019-10-17T20:45:26.993 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000420533 s, 1.2 MB/s 2019-10-17T20:45:26.998 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-10-17T20:45:26.998 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-10-17T20:45:26.998 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-10-17T20:45:26.998 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:26.998 INFO:tasks.ceph:Generating config... 2019-10-17T20:45:26.999 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2019-10-17T20:45:27.000 INFO:tasks.ceph:[global] ms type = async 2019-10-17T20:45:27.000 INFO:tasks.ceph:[global] mon client directed command retry = 5 2019-10-17T20:45:27.000 INFO:tasks.ceph:[mon] debug mon = 20 2019-10-17T20:45:27.000 INFO:tasks.ceph:[mon] debug paxos = 20 2019-10-17T20:45:27.000 INFO:tasks.ceph:[mon] mon osd initial require min compat client = luminous 2019-10-17T20:45:27.000 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2019-10-17T20:45:27.000 INFO:tasks.ceph:[mon] debug ms = 1 2019-10-17T20:45:27.000 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2019-10-17T20:45:27.001 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2019-10-17T20:45:27.001 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2019-10-17T20:45:27.001 INFO:tasks.ceph:[mon] paxos service trim min = 10 2019-10-17T20:45:27.001 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2019-10-17T20:45:27.001 INFO:tasks.ceph:[osd] debug ms = 1 2019-10-17T20:45:27.001 INFO:tasks.ceph:[osd] debug journal = 20 2019-10-17T20:45:27.001 INFO:tasks.ceph:[osd] osd_discard_disconnected_ops = False 2019-10-17T20:45:27.001 INFO:tasks.ceph:[osd] debug osd = 25 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] osd objectstore = filestore 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] osd op queue = debug_random 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] osd sloppy crc = True 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] debug filestore = 20 2019-10-17T20:45:27.002 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2019-10-17T20:45:27.003 INFO:tasks.ceph:Setting up mon.a... 2019-10-17T20:45:27.003 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.003 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-10-17T20:45:27.038 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.keyring 2019-10-17T20:45:27.040 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.040 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-10-17T20:45:27.139 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.139 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-17T20:45:27.222 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.92'), ('mon.c', '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]'), ('mon.b', '172.21.15.204')] 2019-10-17T20:45:27.222 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.223 INFO:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.92 --addv c '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]' --add b 172.21.15.204 --print /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:27.322 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:27.322 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: generated fsid b8a78ffc-fd6b-49fb-a36f-703403180d29 2019-10-17T20:45:27.322 INFO:teuthology.orchestra.run.smithi092.stdout:epoch 0 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:fsid b8a78ffc-fd6b-49fb-a36f-703403180d29 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:last_changed 2019-10-17T20:45:27.324673+0000 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:created 2019-10-17T20:45:27.324673+0000 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:min_mon_release 0 (unknown) 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.b 2019-10-17T20:45:27.323 INFO:teuthology.orchestra.run.smithi092.stdout:2: [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon.c 2019-10-17T20:45:27.324 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-10-17T20:45:27.325 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b8a78ffc-fd6b-49fb-a36f-703403180d29... 2019-10-17T20:45:27.325 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.327 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-17T20:45:27.395 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:27.396 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-10-17T20:45:27.452 INFO:tasks.ceph:Creating admin key on mon.a... 2019-10-17T20:45:27.453 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.453 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-10-17T20:45:27.494 INFO:tasks.ceph:Copying monmap to all nodes... 2019-10-17T20:45:27.566 DEBUG:teuthology.orchestra.remote:smithi092:/etc/ceph/ceph.keyring is 206B 2019-10-17T20:45:27.586 DEBUG:teuthology.orchestra.remote:smithi092:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-10-17T20:45:27.597 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:45:27.597 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.597 INFO:teuthology.orchestra.run.smithi092:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-17T20:45:27.654 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.655 INFO:teuthology.orchestra.run.smithi092:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:27.768 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:45:27.768 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:27.768 INFO:teuthology.orchestra.run.smithi204:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-10-17T20:45:27.825 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:27.825 INFO:teuthology.orchestra.run.smithi204:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:27.947 INFO:tasks.ceph:Setting up mon nodes... 2019-10-17T20:45:27.947 INFO:tasks.ceph:Setting up mgr nodes... 2019-10-17T20:45:27.947 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:27.948 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-10-17T20:45:27.988 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-10-17T20:45:27.991 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:27.991 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-10-17T20:45:28.035 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-10-17T20:45:28.037 INFO:tasks.ceph:Setting up mds nodes... 2019-10-17T20:45:28.037 INFO:tasks.ceph_client:Setting up client nodes... 2019-10-17T20:45:28.038 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.038 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-10-17T20:45:28.073 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-10-17T20:45:28.094 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:28.094 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2019-10-17T20:45:28.128 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/ceph.client.1.keyring 2019-10-17T20:45:28.140 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-10-17T20:45:28.141 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}} 2019-10-17T20:45:28.141 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.141 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-10-17T20:45:28.158 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:28.158 INFO:tasks.ceph:{} 2019-10-17T20:45:28.158 INFO:tasks.ceph:osd.0 2019-10-17T20:45:28.158 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:45:28.158 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.159 INFO:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-10-17T20:45:28.699 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:28.700 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:28.700 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:28.700 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:28.700 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:28.701 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:28.701 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:28.701 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:28.701 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:28.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2019-10-17T20:45:28.703 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.703 INFO:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2019-10-17T20:45:28.769 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.769 INFO:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2019-10-17T20:45:28.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:28.782 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.782 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-10-17T20:45:28.862 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:28.863 INFO:tasks.ceph:{} 2019-10-17T20:45:28.863 INFO:tasks.ceph:osd.1 2019-10-17T20:45:28.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:45:28.863 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:28.863 INFO:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-10-17T20:45:29.425 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:29.425 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:29.426 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:29.426 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:29.426 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:29.426 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:29.426 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:29.426 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:29.427 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:29.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2019-10-17T20:45:29.428 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:29.428 INFO:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2019-10-17T20:45:29.466 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:29.466 INFO:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2019-10-17T20:45:29.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:29.539 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:29.539 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-10-17T20:45:29.614 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:29.615 INFO:tasks.ceph:{} 2019-10-17T20:45:29.615 INFO:tasks.ceph:osd.2 2019-10-17T20:45:29.615 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:45:29.615 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:29.615 INFO:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-10-17T20:45:30.169 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:30.169 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:30.169 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:30.170 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:30.170 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:30.170 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:30.170 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:30.170 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:30.170 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:30.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2019-10-17T20:45:30.171 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:30.172 INFO:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2019-10-17T20:45:30.215 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:30.216 INFO:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2019-10-17T20:45:30.292 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:30.293 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:30.293 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-10-17T20:45:30.372 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:30.373 INFO:tasks.ceph:{} 2019-10-17T20:45:30.373 INFO:tasks.ceph:osd.3 2019-10-17T20:45:30.373 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2019-10-17T20:45:30.373 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:30.373 INFO:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-10-17T20:45:30.945 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:30.946 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:30.947 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:30.947 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2019-10-17T20:45:30.948 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:30.948 INFO:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2019-10-17T20:45:31.016 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:31.016 INFO:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2019-10-17T20:45:31.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:31.030 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:31.030 INFO:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:31.166 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.162+0000 7f5c4473cd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-10-17T20:45:31.168 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.166+0000 7f5c4473cd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-10-17T20:45:31.207 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.202+0000 7f5c4473cd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:31.207 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.206+0000 7f5c4473cd80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:31.459 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.458+0000 7f5c4473cd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:31.461 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.458+0000 7f5c4473cd80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:31.461 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.458+0000 7f5c4473cd80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:31.492 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:31.492 INFO:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-10-17T20:45:31.506 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:31.506 INFO:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:31.607 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.606+0000 7f6d34ac5d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-10-17T20:45:31.608 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.606+0000 7f6d34ac5d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-10-17T20:45:31.653 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.650+0000 7f6d34ac5d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:31.654 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.650+0000 7f6d34ac5d80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:31.810 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.806+0000 7f6d34ac5d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:31.811 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.810+0000 7f6d34ac5d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:31.811 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:31.810+0000 7f6d34ac5d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:31.819 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:31.819 INFO:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-10-17T20:45:31.899 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:31.899 INFO:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:32.008 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.006+0000 7ffa9f865d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-10-17T20:45:32.008 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.006+0000 7ffa9f865d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-10-17T20:45:32.023 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.022+0000 7ffa9f865d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:32.023 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.022+0000 7ffa9f865d80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:32.158 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.154+0000 7ffa9f865d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:32.159 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.154+0000 7ffa9f865d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:32.159 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.154+0000 7ffa9f865d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:32.167 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:32.167 INFO:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-10-17T20:45:32.248 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:32.248 INFO:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:32.349 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.346+0000 7fdcfa232d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-10-17T20:45:32.349 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.346+0000 7fdcfa232d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-10-17T20:45:32.364 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.362+0000 7fdcfa232d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:32.364 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.362+0000 7fdcfa232d80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:32.501 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.498+0000 7fdcfa232d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:32.504 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.502+0000 7fdcfa232d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:32.504 INFO:teuthology.orchestra.run.smithi092.stderr:2019-10-17T20:45:32.502+0000 7fdcfa232d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:32.512 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:32.512 INFO:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-10-17T20:45:32.593 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:32.593 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-10-17T20:45:32.610 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:32.610 INFO:tasks.ceph:{} 2019-10-17T20:45:32.610 INFO:tasks.ceph:osd.4 2019-10-17T20:45:32.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:45:32.610 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:32.610 INFO:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-10-17T20:45:33.150 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:33.150 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:33.150 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:33.151 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:33.151 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:33.151 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:33.151 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:33.151 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:33.151 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:33.152 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2019-10-17T20:45:33.153 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.153 INFO:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2019-10-17T20:45:33.229 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.229 INFO:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2019-10-17T20:45:33.242 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:33.243 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.243 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-10-17T20:45:33.323 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:33.323 INFO:tasks.ceph:{} 2019-10-17T20:45:33.323 INFO:tasks.ceph:osd.5 2019-10-17T20:45:33.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:45:33.324 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.324 INFO:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-10-17T20:45:33.868 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:33.869 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:33.869 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:33.869 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:33.869 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:33.869 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:33.869 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:33.870 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:33.870 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:33.871 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2019-10-17T20:45:33.871 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.871 INFO:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2019-10-17T20:45:33.912 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.913 INFO:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2019-10-17T20:45:33.989 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:33.990 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:33.991 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-10-17T20:45:34.074 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:34.074 INFO:tasks.ceph:{} 2019-10-17T20:45:34.074 INFO:tasks.ceph:osd.6 2019-10-17T20:45:34.074 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:45:34.074 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:34.074 INFO:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-10-17T20:45:34.620 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:34.620 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:34.620 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:34.620 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:34.621 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:34.621 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:34.621 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:34.621 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:34.621 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:34.622 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2019-10-17T20:45:34.622 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:34.622 INFO:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2019-10-17T20:45:34.673 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:34.673 INFO:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2019-10-17T20:45:34.685 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:34.686 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:34.686 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-10-17T20:45:34.766 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-10-17T20:45:34.766 INFO:tasks.ceph:{} 2019-10-17T20:45:34.767 INFO:tasks.ceph:osd.7 2019-10-17T20:45:34.767 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2019-10-17T20:45:34.767 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:34.767 INFO:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-10-17T20:45:35.340 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-10-17T20:45:35.340 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2019-10-17T20:45:35.340 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2019-10-17T20:45:35.340 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-10-17T20:45:35.340 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2019-10-17T20:45:35.341 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-10-17T20:45:35.341 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-10-17T20:45:35.341 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-10-17T20:45:35.341 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-10-17T20:45:35.342 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2019-10-17T20:45:35.342 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:35.342 INFO:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2019-10-17T20:45:35.422 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:35.422 INFO:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2019-10-17T20:45:35.435 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: /sbin/restorecon: command not found 2019-10-17T20:45:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-10-17T20:45:35.437 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:35.437 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:35.571 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.567+0000 7fc569989d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-10-17T20:45:35.573 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.567+0000 7fc569989d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-10-17T20:45:35.616 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.611+0000 7fc569989d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:35.618 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.615+0000 7fc569989d80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:35.877 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.871+0000 7fc569989d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:35.879 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.875+0000 7fc569989d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:35.879 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:35.875+0000 7fc569989d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:35.900 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:35.901 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-10-17T20:45:35.916 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:35.916 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:36.018 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.015+0000 7fccf57b2d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-10-17T20:45:36.019 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.015+0000 7fccf57b2d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-10-17T20:45:36.060 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.055+0000 7fccf57b2d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:36.062 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.059+0000 7fccf57b2d80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:36.236 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.231+0000 7fccf57b2d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:36.237 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.231+0000 7fccf57b2d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:36.237 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.231+0000 7fccf57b2d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:36.245 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:36.245 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-10-17T20:45:36.326 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:36.327 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:36.430 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.427+0000 7f7201205d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-10-17T20:45:36.430 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.427+0000 7f7201205d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-10-17T20:45:36.446 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.443+0000 7f7201205d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:36.447 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.443+0000 7f7201205d80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:36.576 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.571+0000 7f7201205d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:36.577 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.571+0000 7f7201205d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:36.577 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.571+0000 7f7201205d80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:36.586 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:36.586 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-10-17T20:45:36.666 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:36.667 INFO:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:36.771 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.767+0000 7fca46e2dd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-10-17T20:45:36.771 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.767+0000 7fca46e2dd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-10-17T20:45:36.786 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.783+0000 7fca46e2dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:36.786 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.783+0000 7fca46e2dd80 -1 journal FileJournal::_open_file : zeroing journal 2019-10-17T20:45:36.922 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.919+0000 7fca46e2dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:36.923 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.919+0000 7fca46e2dd80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:36.923 INFO:teuthology.orchestra.run.smithi204.stderr:2019-10-17T20:45:36.919+0000 7fca46e2dd80 -1 journal do_read_entry(4096): bad header magic 2019-10-17T20:45:36.933 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:36.933 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-10-17T20:45:37.013 INFO:tasks.ceph:Reading keys from all nodes... 2019-10-17T20:45:37.014 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.014 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:37.021 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.NLXH9yTJr8 2019-10-17T20:45:37.021 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.022 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.NLXH9yTJr8 2019-10-17T20:45:37.102 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.102 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.NLXH9yTJr8 2019-10-17T20:45:37.257 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.NLXH9yTJr8 is 56B 2019-10-17T20:45:37.270 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.270 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.NLXH9yTJr8 2019-10-17T20:45:37.278 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.278 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:37.349 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.tgZxhYATfB 2019-10-17T20:45:37.349 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.349 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.tgZxhYATfB 2019-10-17T20:45:37.429 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.429 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.tgZxhYATfB 2019-10-17T20:45:37.587 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.tgZxhYATfB is 56B 2019-10-17T20:45:37.599 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.599 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.tgZxhYATfB 2019-10-17T20:45:37.606 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.606 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:37.677 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.6C60I8hXFY 2019-10-17T20:45:37.677 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.678 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.6C60I8hXFY 2019-10-17T20:45:37.758 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.758 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.6C60I8hXFY 2019-10-17T20:45:37.912 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.6C60I8hXFY is 56B 2019-10-17T20:45:37.925 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.925 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.6C60I8hXFY 2019-10-17T20:45:37.933 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:37.933 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:38.003 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.OraBGm7v4R 2019-10-17T20:45:38.003 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.003 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.OraBGm7v4R 2019-10-17T20:45:38.084 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.084 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.OraBGm7v4R 2019-10-17T20:45:38.251 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.OraBGm7v4R is 56B 2019-10-17T20:45:38.267 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.267 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.OraBGm7v4R 2019-10-17T20:45:38.273 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.273 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:38.345 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.BPGwI7Vjzf 2019-10-17T20:45:38.345 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.345 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.BPGwI7Vjzf 2019-10-17T20:45:38.425 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.425 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.BPGwI7Vjzf 2019-10-17T20:45:38.577 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.BPGwI7Vjzf is 56B 2019-10-17T20:45:38.590 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:38.590 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.BPGwI7Vjzf 2019-10-17T20:45:38.598 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:38.598 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:38.605 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.uzgZTm4Enf 2019-10-17T20:45:38.605 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:38.605 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.uzgZTm4Enf 2019-10-17T20:45:38.686 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:38.686 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.uzgZTm4Enf 2019-10-17T20:45:38.836 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.uzgZTm4Enf is 56B 2019-10-17T20:45:38.847 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:38.847 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.uzgZTm4Enf 2019-10-17T20:45:38.855 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:38.855 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:38.925 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.I8QomQCM8p 2019-10-17T20:45:38.925 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:38.926 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.I8QomQCM8p 2019-10-17T20:45:39.005 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.005 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.I8QomQCM8p 2019-10-17T20:45:39.160 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.I8QomQCM8p is 56B 2019-10-17T20:45:39.171 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.172 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.I8QomQCM8p 2019-10-17T20:45:39.180 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.180 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:39.250 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.idEjCwDDtx 2019-10-17T20:45:39.251 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.251 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.idEjCwDDtx 2019-10-17T20:45:39.330 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.330 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.idEjCwDDtx 2019-10-17T20:45:39.482 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.idEjCwDDtx is 56B 2019-10-17T20:45:39.495 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.495 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.idEjCwDDtx 2019-10-17T20:45:39.502 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.502 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:39.573 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.nhYvhUHn4r 2019-10-17T20:45:39.574 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.574 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.nhYvhUHn4r 2019-10-17T20:45:39.654 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.654 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.nhYvhUHn4r 2019-10-17T20:45:39.806 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.nhYvhUHn4r is 56B 2019-10-17T20:45:39.818 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.818 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.nhYvhUHn4r 2019-10-17T20:45:39.825 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.825 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:39.896 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.0f1qPRFuK1 2019-10-17T20:45:39.896 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.896 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.0f1qPRFuK1 2019-10-17T20:45:39.976 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:39.976 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.0f1qPRFuK1 2019-10-17T20:45:40.128 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.0f1qPRFuK1 is 56B 2019-10-17T20:45:40.139 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.140 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.0f1qPRFuK1 2019-10-17T20:45:40.156 DEBUG:teuthology.orchestra.remote:smithi092:/etc/ceph/ceph.client.0.keyring is 59B 2019-10-17T20:45:40.208 DEBUG:teuthology.orchestra.remote:smithi204:/etc/ceph/ceph.client.1.keyring is 59B 2019-10-17T20:45:40.222 INFO:tasks.ceph:Adding keys to all mons... 2019-10-17T20:45:40.222 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.222 INFO:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-17T20:45:40.228 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.228 INFO:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2019-10-17T20:45:40.273 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.y] 2019-10-17T20:45:40.274 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDn0qhdvgQROxAAP8ZHi7vh1nu3oe9+zUq1ug== 2019-10-17T20:45:40.274 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2019-10-17T20:45:40.274 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDr0qhdhxIRChAA1KzE81u8wGvangIxvX5hEw== 2019-10-17T20:45:40.274 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2019-10-17T20:45:40.275 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDr0qhdJ+xfJBAA/VNOAR/+MoE6wf2Ni6NyUg== 2019-10-17T20:45:40.275 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2019-10-17T20:45:40.275 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDs0qhdRXmlABAABTXXDrcUXlbdevFqrEPQ/w== 2019-10-17T20:45:40.275 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2019-10-17T20:45:40.276 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDs0qhdeRX8FBAAieWVkcR7k1pxRtiVIXQP+g== 2019-10-17T20:45:40.276 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2019-10-17T20:45:40.276 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDo0qhdgu8cAhAAXliq47m/aSUWR0O1QKaAAQ== 2019-10-17T20:45:40.276 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2019-10-17T20:45:40.277 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDv0qhdieITIhAAfb14cjfONNbsdmCBwNxJ4A== 2019-10-17T20:45:40.277 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2019-10-17T20:45:40.277 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDw0qhdePYbARAAY52ACuncZ+OclmgofYGzkw== 2019-10-17T20:45:40.278 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.6] 2019-10-17T20:45:40.278 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDw0qhdt82nGRAAnWZPfzETtNdY0n2Jt5Qg3Q== 2019-10-17T20:45:40.278 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.7] 2019-10-17T20:45:40.278 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDw0qhd+9j0LRAAZTjuBzMsILO2WggounMJfg== 2019-10-17T20:45:40.278 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2019-10-17T20:45:40.279 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDo0qhdAXKGBBAANCNGN8i6g2dt8g26CdO6mg== 2019-10-17T20:45:40.279 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2019-10-17T20:45:40.279 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDo0qhdx1yzBxAA17XhHsqqengKbp9wM8t1JQ== 2019-10-17T20:45:40.281 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2019-10-17T20:45:40.282 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDn0qhdvgQROxAAP8ZHi7vh1nu3oe9+zUq1ug== 2019-10-17T20:45:40.282 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2019-10-17T20:45:40.282 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDr0qhdhxIRChAA1KzE81u8wGvangIxvX5hEw== 2019-10-17T20:45:40.283 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2019-10-17T20:45:40.283 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDr0qhdJ+xfJBAA/VNOAR/+MoE6wf2Ni6NyUg== 2019-10-17T20:45:40.283 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2019-10-17T20:45:40.283 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDs0qhdRXmlABAABTXXDrcUXlbdevFqrEPQ/w== 2019-10-17T20:45:40.284 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2019-10-17T20:45:40.284 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDs0qhdeRX8FBAAieWVkcR7k1pxRtiVIXQP+g== 2019-10-17T20:45:40.284 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2019-10-17T20:45:40.284 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDo0qhdgu8cAhAAXliq47m/aSUWR0O1QKaAAQ== 2019-10-17T20:45:40.284 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2019-10-17T20:45:40.285 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDv0qhdieITIhAAfb14cjfONNbsdmCBwNxJ4A== 2019-10-17T20:45:40.285 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2019-10-17T20:45:40.285 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDw0qhdePYbARAAY52ACuncZ+OclmgofYGzkw== 2019-10-17T20:45:40.286 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2019-10-17T20:45:40.286 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDw0qhdt82nGRAAnWZPfzETtNdY0n2Jt5Qg3Q== 2019-10-17T20:45:40.286 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2019-10-17T20:45:40.286 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDw0qhd+9j0LRAAZTjuBzMsILO2WggounMJfg== 2019-10-17T20:45:40.286 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2019-10-17T20:45:40.287 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDo0qhdAXKGBBAANCNGN8i6g2dt8g26CdO6mg== 2019-10-17T20:45:40.287 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2019-10-17T20:45:40.287 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDo0qhdx1yzBxAA17XhHsqqengKbp9wM8t1JQ== 2019-10-17T20:45:40.288 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.288 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-17T20:45:40.355 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.356 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-17T20:45:40.398 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.399 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.465 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.465 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.509 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.510 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.577 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.577 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.621 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.621 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.688 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.688 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.730 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.731 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.797 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.797 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:40.841 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.841 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-17T20:45:40.908 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:40.908 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-10-17T20:45:40.951 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:40.951 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.024 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:41.024 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.073 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.073 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.140 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:41.140 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.185 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.185 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.252 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:41.253 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.296 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.296 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.363 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:41.363 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-10-17T20:45:41.406 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.406 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-17T20:45:41.475 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:41.475 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-17T20:45:41.521 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.521 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-17T20:45:41.590 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:41.590 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-10-17T20:45:41.635 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-10-17T20:45:41.636 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.636 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-10-17T20:45:41.716 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.716 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-17T20:45:41.857 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.857 INFO:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-10-17T20:45:41.872 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.873 INFO:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-10-17T20:45:41.951 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:41.952 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-17T20:45:42.092 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.092 INFO:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-10-17T20:45:42.107 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.108 INFO:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-10-17T20:45:42.124 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.124 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-10-17T20:45:42.248 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.248 INFO:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-10-17T20:45:42.262 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.262 INFO:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:42.265 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.265 INFO:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-10-17T20:45:42.338 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-10-17T20:45:42.338 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.338 INFO:teuthology.orchestra.run.smithi092:> which systemctl 2019-10-17T20:45:42.346 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/systemctl 2019-10-17T20:45:42.346 INFO:tasks.ceph.mon.a:Restarting daemon 2019-10-17T20:45:42.346 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.346 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-10-17T20:45:42.413 INFO:tasks.ceph.mon.a:Started 2019-10-17T20:45:42.413 INFO:tasks.ceph.mon.c:Restarting daemon 2019-10-17T20:45:42.413 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.413 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-10-17T20:45:42.416 INFO:tasks.ceph.mon.c:Started 2019-10-17T20:45:42.417 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.417 INFO:teuthology.orchestra.run.smithi204:> which systemctl 2019-10-17T20:45:42.424 INFO:teuthology.orchestra.run.smithi204.stdout:/bin/systemctl 2019-10-17T20:45:42.424 INFO:tasks.ceph.mon.b:Restarting daemon 2019-10-17T20:45:42.425 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.425 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-10-17T20:45:42.494 INFO:tasks.ceph.mon.b:Started 2019-10-17T20:45:42.494 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-10-17T20:45:42.495 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-10-17T20:45:42.495 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.495 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2019-10-17T20:45:42.499 INFO:tasks.ceph.mgr.y:Started 2019-10-17T20:45:42.499 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-10-17T20:45:42.499 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:42.499 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-10-17T20:45:42.504 INFO:tasks.ceph.mgr.x:Started 2019-10-17T20:45:42.505 INFO:tasks.ceph:Setting crush tunables to optimal 2019-10-17T20:45:42.505 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:42.505 INFO:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd crush tunables optimal 2019-10-17T20:45:47.943 INFO:teuthology.orchestra.run.smithi092.stderr:adjusted tunables profile to optimal 2019-10-17T20:45:47.956 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-10-17T20:45:47.957 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:47.957 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:47.964 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.aBp6gF3vP2 2019-10-17T20:45:47.964 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:47.964 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.aBp6gF3vP2 2019-10-17T20:45:48.044 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.044 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.aBp6gF3vP2 2019-10-17T20:45:48.194 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.aBp6gF3vP2 is 37B 2019-10-17T20:45:48.207 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.207 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.aBp6gF3vP2 2019-10-17T20:45:48.214 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.214 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:48.285 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.MI34GIR81F 2019-10-17T20:45:48.285 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.286 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.MI34GIR81F 2019-10-17T20:45:48.365 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.365 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.MI34GIR81F 2019-10-17T20:45:48.518 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.MI34GIR81F is 37B 2019-10-17T20:45:48.529 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.529 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.MI34GIR81F 2019-10-17T20:45:48.538 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.538 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:48.609 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.JYU1BAIfdQ 2019-10-17T20:45:48.609 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.609 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.JYU1BAIfdQ 2019-10-17T20:45:48.689 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.689 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.JYU1BAIfdQ 2019-10-17T20:45:48.839 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.JYU1BAIfdQ is 37B 2019-10-17T20:45:48.850 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.850 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.JYU1BAIfdQ 2019-10-17T20:45:48.857 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.858 INFO:teuthology.orchestra.run.smithi092:> mktemp 2019-10-17T20:45:48.930 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.htEnjrICY4 2019-10-17T20:45:48.930 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:48.931 INFO:teuthology.orchestra.run.smithi092:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.htEnjrICY4 2019-10-17T20:45:49.012 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:49.012 INFO:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.htEnjrICY4 2019-10-17T20:45:49.162 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.htEnjrICY4 is 37B 2019-10-17T20:45:49.173 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:49.173 INFO:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.htEnjrICY4 2019-10-17T20:45:49.181 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.182 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:49.189 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.UgCOfRcVSZ 2019-10-17T20:45:49.189 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.189 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.UgCOfRcVSZ 2019-10-17T20:45:49.269 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.269 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.UgCOfRcVSZ 2019-10-17T20:45:49.422 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.UgCOfRcVSZ is 37B 2019-10-17T20:45:49.435 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.435 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.UgCOfRcVSZ 2019-10-17T20:45:49.443 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.443 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:49.513 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.oFBvSxOOPL 2019-10-17T20:45:49.513 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.513 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.oFBvSxOOPL 2019-10-17T20:45:49.593 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.594 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.oFBvSxOOPL 2019-10-17T20:45:49.744 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.oFBvSxOOPL is 37B 2019-10-17T20:45:49.754 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.755 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.oFBvSxOOPL 2019-10-17T20:45:49.762 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.762 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:49.834 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.YTO6k2blwE 2019-10-17T20:45:49.834 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.834 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.YTO6k2blwE 2019-10-17T20:45:49.913 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:49.913 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.YTO6k2blwE 2019-10-17T20:45:50.067 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.YTO6k2blwE is 37B 2019-10-17T20:45:50.118 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.118 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.YTO6k2blwE 2019-10-17T20:45:50.125 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.125 INFO:teuthology.orchestra.run.smithi204:> mktemp 2019-10-17T20:45:50.196 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.yeKDrwWwAO 2019-10-17T20:45:50.196 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.196 INFO:teuthology.orchestra.run.smithi204:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.yeKDrwWwAO 2019-10-17T20:45:50.276 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.276 INFO:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.yeKDrwWwAO 2019-10-17T20:45:50.428 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.yeKDrwWwAO is 37B 2019-10-17T20:45:50.439 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.440 INFO:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.yeKDrwWwAO 2019-10-17T20:45:50.447 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.447 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 8ab85492-12c6-4c7c-8201-f7659d0d84ec 0 2019-10-17T20:45:50.749 INFO:teuthology.orchestra.run.smithi204.stdout:0 2019-10-17T20:45:50.766 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:50.766 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new be05be4d-57ae-43bf-ac11-3341e5cabe9b 1 2019-10-17T20:45:50.989 INFO:teuthology.orchestra.run.smithi204.stdout:1 2019-10-17T20:45:51.002 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:51.003 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 9cbc29b0-9860-4f41-a3bb-aafecd59daba 2 2019-10-17T20:45:51.299 INFO:teuthology.orchestra.run.smithi204.stdout:2 2019-10-17T20:45:51.313 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:51.313 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 110aa2aa-d522-4c91-aa4f-76ab73f79d7a 3 2019-10-17T20:45:51.598 INFO:teuthology.orchestra.run.smithi204.stdout:3 2019-10-17T20:45:51.613 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:51.613 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 90b71d59-60f5-49b5-947c-fb012318f4eb 4 2019-10-17T20:45:51.897 INFO:teuthology.orchestra.run.smithi204.stdout:4 2019-10-17T20:45:51.913 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:51.914 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 7b3c3b9e-f75e-49e8-b195-ed56a924a36d 5 2019-10-17T20:45:52.200 INFO:teuthology.orchestra.run.smithi204.stdout:5 2019-10-17T20:45:52.215 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:52.215 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 09d4f546-1e94-43f6-a76a-98cb6b9fffd5 6 2019-10-17T20:45:52.518 INFO:teuthology.orchestra.run.smithi204.stdout:6 2019-10-17T20:45:52.534 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:52.534 INFO:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 131feb5d-5cde-41b8-83bc-8a8258beed6a 7 2019-10-17T20:45:52.817 INFO:teuthology.orchestra.run.smithi204.stdout:7 2019-10-17T20:45:52.832 INFO:tasks.ceph.osd.0:Restarting daemon 2019-10-17T20:45:52.833 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:52.833 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-10-17T20:45:52.837 INFO:tasks.ceph.osd.0:Started 2019-10-17T20:45:52.837 INFO:tasks.ceph.osd.1:Restarting daemon 2019-10-17T20:45:52.837 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:52.837 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-10-17T20:45:52.841 INFO:tasks.ceph.osd.1:Started 2019-10-17T20:45:52.841 INFO:tasks.ceph.osd.2:Restarting daemon 2019-10-17T20:45:52.841 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:52.841 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-10-17T20:45:52.844 INFO:tasks.ceph.osd.2:Started 2019-10-17T20:45:52.845 INFO:tasks.ceph.osd.3:Restarting daemon 2019-10-17T20:45:52.845 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:52.845 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-10-17T20:45:52.848 INFO:tasks.ceph.osd.3:Started 2019-10-17T20:45:52.848 INFO:tasks.ceph.osd.4:Restarting daemon 2019-10-17T20:45:52.848 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:52.849 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-10-17T20:45:52.852 INFO:tasks.ceph.osd.4:Started 2019-10-17T20:45:52.852 INFO:tasks.ceph.osd.5:Restarting daemon 2019-10-17T20:45:52.852 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:52.853 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-10-17T20:45:52.856 INFO:tasks.ceph.osd.5:Started 2019-10-17T20:45:52.856 INFO:tasks.ceph.osd.6:Restarting daemon 2019-10-17T20:45:52.856 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:52.857 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2019-10-17T20:45:52.860 INFO:tasks.ceph.osd.6:Started 2019-10-17T20:45:52.860 INFO:tasks.ceph.osd.7:Restarting daemon 2019-10-17T20:45:52.860 INFO:teuthology.orchestra.run.smithi204:Running: 2019-10-17T20:45:52.861 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2019-10-17T20:45:52.863 INFO:tasks.ceph.osd.7:Started 2019-10-17T20:45:52.864 INFO:tasks.ceph:Waiting for OSDs to come up 2019-10-17T20:45:52.894 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T20:45:52.894+0000 7ff3ad06fd80 -1 Falling back to public interface 2019-10-17T20:45:52.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:45:52.898+0000 7fd47803fd80 -1 Falling back to public interface 2019-10-17T20:45:52.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T20:45:52.902+0000 7fb647e5cd80 -1 Falling back to public interface 2019-10-17T20:45:52.906 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:45:52.902+0000 7f65400f6d80 -1 Falling back to public interface 2019-10-17T20:45:52.909 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T20:45:52.906+0000 7ff3ad06fd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.909 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T20:45:52.906+0000 7ff3ad06fd80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.909 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T20:45:52.906+0000 7ff3ad06fd80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.913 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:45:52.911+0000 7efdf5cc8d80 -1 Falling back to public interface 2019-10-17T20:45:52.914 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T20:45:52.914+0000 7ff3ad06fd80 -1 osd.3 0 log_to_monitors {default=true} 2019-10-17T20:45:52.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T20:45:52.911+0000 7f5526a39d80 -1 Falling back to public interface 2019-10-17T20:45:52.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T20:45:52.915+0000 7f4529cf1d80 -1 Falling back to public interface 2019-10-17T20:45:52.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:45:52.918+0000 7fd47803fd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:45:52.918+0000 7fd47803fd80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:45:52.918+0000 7fd47803fd80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.921 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:45:52.915+0000 7f2308f75d80 -1 Falling back to public interface 2019-10-17T20:45:52.923 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T20:45:52.922+0000 7fb647e5cd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.923 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T20:45:52.922+0000 7fb647e5cd80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.923 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T20:45:52.922+0000 7fb647e5cd80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:45:52.922+0000 7fd47803fd80 -1 osd.1 0 log_to_monitors {default=true} 2019-10-17T20:45:52.926 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:45:52.922+0000 7f65400f6d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.926 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:45:52.926+0000 7f65400f6d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.926 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:45:52.926+0000 7f65400f6d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.927 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:45:52.923+0000 7efdf5cc8d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.927 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:45:52.923+0000 7efdf5cc8d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.927 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:45:52.923+0000 7efdf5cc8d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.928 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T20:45:52.926+0000 7fb647e5cd80 -1 osd.0 0 log_to_monitors {default=true} 2019-10-17T20:45:52.931 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:45:52.930+0000 7f65400f6d80 -1 osd.2 0 log_to_monitors {default=true} 2019-10-17T20:45:52.933 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:45:52.927+0000 7efdf5cc8d80 -1 osd.4 0 log_to_monitors {default=true} 2019-10-17T20:45:52.934 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T20:45:52.931+0000 7f5526a39d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T20:45:52.931+0000 7f5526a39d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T20:45:52.931+0000 7f5526a39d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T20:45:52.931+0000 7f4529cf1d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T20:45:52.931+0000 7f4529cf1d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T20:45:52.931+0000 7f4529cf1d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.939 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T20:45:52.935+0000 7f4529cf1d80 -1 osd.5 0 log_to_monitors {default=true} 2019-10-17T20:45:52.940 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:45:52.935+0000 7f2308f75d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2019-10-17T20:45:52.940 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:45:52.935+0000 7f2308f75d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.940 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:45:52.935+0000 7f2308f75d80 -1 journal do_read_entry(8192): bad header magic 2019-10-17T20:45:52.940 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T20:45:52.935+0000 7f5526a39d80 -1 osd.7 0 log_to_monitors {default=true} 2019-10-17T20:45:52.944 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:45:52.939+0000 7f2308f75d80 -1 osd.6 0 log_to_monitors {default=true} 2019-10-17T20:45:53.672 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:45:53.672 INFO:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-17T20:45:53.958 INFO:teuthology.misc.health.smithi092.stdout: 2019-10-17T20:45:53.958 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":11,"fsid":"b8a78ffc-fd6b-49fb-a36f-703403180d29","created":"2019-10-17T20:45:47.750284+0000","modified":"2019-10-17T20:45:53.809213+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"8ab85492-12c6-4c7c-8201-f7659d0d84ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be05be4d-57ae-43bf-ac11-3341e5cabe9b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cbc29b0-9860-4f41-a3bb-aafecd59daba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"110aa2aa-d522-4c91-aa4f-76ab73f79d7a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"90b71d59-60f5-49b5-947c-fb012318f4eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7b3c3b9e-f75e-49e8-b195-ed56a924a36d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"09d4f546-1e94-43f6-a76a-98cb6b9fffd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"131feb5d-5cde-41b8-83bc-8a8258beed6a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-17T20:45:53.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-10-17T20:45:54.823 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:45:54.819+0000 7f22e89bf700 -1 osd.6 0 waiting for initial osdmap 2019-10-17T20:45:54.824 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T20:45:54.819+0000 7f450973b700 -1 osd.5 0 waiting for initial osdmap 2019-10-17T20:45:54.826 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:45:54.823+0000 7efdd5712700 -1 osd.4 0 waiting for initial osdmap 2019-10-17T20:45:54.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:45:54.826+0000 7fd457a89700 -1 osd.1 0 waiting for initial osdmap 2019-10-17T20:45:54.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T20:45:54.823+0000 7f5506483700 -1 osd.7 0 waiting for initial osdmap 2019-10-17T20:45:54.830 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:45:54.826+0000 7f651fb40700 -1 osd.2 0 waiting for initial osdmap 2019-10-17T20:45:54.830 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T20:45:54.826+0000 7fb6278a6700 -1 osd.0 0 waiting for initial osdmap 2019-10-17T20:45:54.831 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T20:45:54.830+0000 7ff38cab9700 -1 osd.3 0 waiting for initial osdmap 2019-10-17T20:46:00.778 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:00.778 INFO:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-17T20:46:01.043 INFO:teuthology.misc.health.smithi092.stdout: 2019-10-17T20:46:01.044 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":14,"fsid":"b8a78ffc-fd6b-49fb-a36f-703403180d29","created":"2019-10-17T20:45:47.750284+0000","modified":"2019-10-17T20:45:56.845144+0000","last_up_change":"2019-10-17T20:45:55.824589+0000","last_in_change":"2019-10-17T20:45:55.824589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"8ab85492-12c6-4c7c-8201-f7659d0d84ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13089},{"type":"v1","addr":"172.21.15.92:6819","nonce":13089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":13089},{"type":"v1","addr":"172.21.15.92:6821","nonce":13089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6827","nonce":13089},{"type":"v1","addr":"172.21.15.92:6828","nonce":13089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13089},{"type":"v1","addr":"172.21.15.92:6825","nonce":13089}]},"public_addr":"172.21.15.92:6819/13089","cluster_addr":"172.21.15.92:6821/13089","heartbeat_back_addr":"172.21.15.92:6828/13089","heartbeat_front_addr":"172.21.15.92:6825/13089","state":["exists","up"]},{"osd":1,"uuid":"be05be4d-57ae-43bf-ac11-3341e5cabe9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":13091},{"type":"v1","addr":"172.21.15.92:6811","nonce":13091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13091},{"type":"v1","addr":"172.21.15.92:6813","nonce":13091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13091},{"type":"v1","addr":"172.21.15.92:6818","nonce":13091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":13091},{"type":"v1","addr":"172.21.15.92:6815","nonce":13091}]},"public_addr":"172.21.15.92:6811/13091","cluster_addr":"172.21.15.92:6813/13091","heartbeat_back_addr":"172.21.15.92:6818/13091","heartbeat_front_addr":"172.21.15.92:6815/13091","state":["exists","up"]},{"osd":2,"uuid":"9cbc29b0-9860-4f41-a3bb-aafecd59daba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13092},{"type":"v1","addr":"172.21.15.92:6824","nonce":13092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":13092},{"type":"v1","addr":"172.21.15.92:6829","nonce":13092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":13092},{"type":"v1","addr":"172.21.15.92:6833","nonce":13092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13092},{"type":"v1","addr":"172.21.15.92:6831","nonce":13092}]},"public_addr":"172.21.15.92:6824/13092","cluster_addr":"172.21.15.92:6829/13092","heartbeat_back_addr":"172.21.15.92:6833/13092","heartbeat_front_addr":"172.21.15.92:6831/13092","state":["exists","up"]},{"osd":3,"uuid":"110aa2aa-d522-4c91-aa4f-76ab73f79d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13090},{"type":"v1","addr":"172.21.15.92:6803","nonce":13090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13090},{"type":"v1","addr":"172.21.15.92:6805","nonce":13090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13090},{"type":"v1","addr":"172.21.15.92:6809","nonce":13090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13090},{"type":"v1","addr":"172.21.15.92:6807","nonce":13090}]},"public_addr":"172.21.15.92:6803/13090","cluster_addr":"172.21.15.92:6805/13090","heartbeat_back_addr":"172.21.15.92:6809/13090","heartbeat_front_addr":"172.21.15.92:6807/13090","state":["exists","up"]},{"osd":4,"uuid":"90b71d59-60f5-49b5-947c-fb012318f4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":13141},{"type":"v1","addr":"172.21.15.204:6801","nonce":13141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13141},{"type":"v1","addr":"172.21.15.204:6803","nonce":13141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13141},{"type":"v1","addr":"172.21.15.204:6807","nonce":13141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13141},{"type":"v1","addr":"172.21.15.204:6805","nonce":13141}]},"public_addr":"172.21.15.204:6801/13141","cluster_addr":"172.21.15.204:6803/13141","heartbeat_back_addr":"172.21.15.204:6807/13141","heartbeat_front_addr":"172.21.15.204:6805/13141","state":["exists","up"]},{"osd":5,"uuid":"7b3c3b9e-f75e-49e8-b195-ed56a924a36d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":13142},{"type":"v1","addr":"172.21.15.204:6817","nonce":13142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13142},{"type":"v1","addr":"172.21.15.204:6819","nonce":13142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13142},{"type":"v1","addr":"172.21.15.204:6824","nonce":13142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13142},{"type":"v1","addr":"172.21.15.204:6821","nonce":13142}]},"public_addr":"172.21.15.204:6817/13142","cluster_addr":"172.21.15.204:6819/13142","heartbeat_back_addr":"172.21.15.204:6824/13142","heartbeat_front_addr":"172.21.15.204:6821/13142","state":["exists","up"]},{"osd":6,"uuid":"09d4f546-1e94-43f6-a76a-98cb6b9fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6823","nonce":13144},{"type":"v1","addr":"172.21.15.204:6825","nonce":13144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13144},{"type":"v1","addr":"172.21.15.204:6827","nonce":13144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13144},{"type":"v1","addr":"172.21.15.204:6831","nonce":13144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13144},{"type":"v1","addr":"172.21.15.204:6829","nonce":13144}]},"public_addr":"172.21.15.204:6825/13144","cluster_addr":"172.21.15.204:6827/13144","heartbeat_back_addr":"172.21.15.204:6831/13144","heartbeat_front_addr":"172.21.15.204:6829/13144","state":["exists","up"]},{"osd":7,"uuid":"131feb5d-5cde-41b8-83bc-8a8258beed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13143},{"type":"v1","addr":"172.21.15.204:6809","nonce":13143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13143},{"type":"v1","addr":"172.21.15.204:6811","nonce":13143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13143},{"type":"v1","addr":"172.21.15.204:6816","nonce":13143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13143},{"type":"v1","addr":"172.21.15.204:6813","nonce":13143}]},"public_addr":"172.21.15.204:6809/13143","cluster_addr":"172.21.15.204:6811/13143","heartbeat_back_addr":"172.21.15.204:6816/13143","heartbeat_front_addr":"172.21.15.204:6813/13143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.968011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.963899+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.947850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.901012+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.943555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.938685+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.908440+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.930129+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-17T20:46:01.054 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-17T20:46:01.055 INFO:tasks.ceph:Creating RBD pool 2019-10-17T20:46:01.055 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:01.055 INFO:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd pool create rbd 8 2019-10-17T20:46:02.144 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'rbd' created 2019-10-17T20:46:02.159 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:02.159 INFO:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-10-17T20:46:03.177 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rbd' on pool 'rbd' 2019-10-17T20:46:03.196 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-10-17T20:46:03.196 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:03.197 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-17T20:46:03.197 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-10-17T20:46:03.469 INFO:teuthology.orchestra.run.smithi092.stdout: 2019-10-17T20:46:03.469 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":16,"fsid":"b8a78ffc-fd6b-49fb-a36f-703403180d29","created":"2019-10-17T20:45:47.750284+0000","modified":"2019-10-17T20:46:03.133487+0000","last_up_change":"2019-10-17T20:45:55.824589+0000","last_in_change":"2019-10-17T20:45:55.824589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-17T20:46:01.367245+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8ab85492-12c6-4c7c-8201-f7659d0d84ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13089},{"type":"v1","addr":"172.21.15.92:6819","nonce":13089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":13089},{"type":"v1","addr":"172.21.15.92:6821","nonce":13089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6827","nonce":13089},{"type":"v1","addr":"172.21.15.92:6828","nonce":13089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13089},{"type":"v1","addr":"172.21.15.92:6825","nonce":13089}]},"public_addr":"172.21.15.92:6819/13089","cluster_addr":"172.21.15.92:6821/13089","heartbeat_back_addr":"172.21.15.92:6828/13089","heartbeat_front_addr":"172.21.15.92:6825/13089","state":["exists","up"]},{"osd":1,"uuid":"be05be4d-57ae-43bf-ac11-3341e5cabe9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":13091},{"type":"v1","addr":"172.21.15.92:6811","nonce":13091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13091},{"type":"v1","addr":"172.21.15.92:6813","nonce":13091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13091},{"type":"v1","addr":"172.21.15.92:6818","nonce":13091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":13091},{"type":"v1","addr":"172.21.15.92:6815","nonce":13091}]},"public_addr":"172.21.15.92:6811/13091","cluster_addr":"172.21.15.92:6813/13091","heartbeat_back_addr":"172.21.15.92:6818/13091","heartbeat_front_addr":"172.21.15.92:6815/13091","state":["exists","up"]},{"osd":2,"uuid":"9cbc29b0-9860-4f41-a3bb-aafecd59daba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13092},{"type":"v1","addr":"172.21.15.92:6824","nonce":13092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":13092},{"type":"v1","addr":"172.21.15.92:6829","nonce":13092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":13092},{"type":"v1","addr":"172.21.15.92:6833","nonce":13092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13092},{"type":"v1","addr":"172.21.15.92:6831","nonce":13092}]},"public_addr":"172.21.15.92:6824/13092","cluster_addr":"172.21.15.92:6829/13092","heartbeat_back_addr":"172.21.15.92:6833/13092","heartbeat_front_addr":"172.21.15.92:6831/13092","state":["exists","up"]},{"osd":3,"uuid":"110aa2aa-d522-4c91-aa4f-76ab73f79d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13090},{"type":"v1","addr":"172.21.15.92:6803","nonce":13090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13090},{"type":"v1","addr":"172.21.15.92:6805","nonce":13090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13090},{"type":"v1","addr":"172.21.15.92:6809","nonce":13090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13090},{"type":"v1","addr":"172.21.15.92:6807","nonce":13090}]},"public_addr":"172.21.15.92:6803/13090","cluster_addr":"172.21.15.92:6805/13090","heartbeat_back_addr":"172.21.15.92:6809/13090","heartbeat_front_addr":"172.21.15.92:6807/13090","state":["exists","up"]},{"osd":4,"uuid":"90b71d59-60f5-49b5-947c-fb012318f4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":13141},{"type":"v1","addr":"172.21.15.204:6801","nonce":13141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13141},{"type":"v1","addr":"172.21.15.204:6803","nonce":13141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13141},{"type":"v1","addr":"172.21.15.204:6807","nonce":13141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13141},{"type":"v1","addr":"172.21.15.204:6805","nonce":13141}]},"public_addr":"172.21.15.204:6801/13141","cluster_addr":"172.21.15.204:6803/13141","heartbeat_back_addr":"172.21.15.204:6807/13141","heartbeat_front_addr":"172.21.15.204:6805/13141","state":["exists","up"]},{"osd":5,"uuid":"7b3c3b9e-f75e-49e8-b195-ed56a924a36d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":13142},{"type":"v1","addr":"172.21.15.204:6817","nonce":13142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13142},{"type":"v1","addr":"172.21.15.204:6819","nonce":13142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13142},{"type":"v1","addr":"172.21.15.204:6824","nonce":13142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13142},{"type":"v1","addr":"172.21.15.204:6821","nonce":13142}]},"public_addr":"172.21.15.204:6817/13142","cluster_addr":"172.21.15.204:6819/13142","heartbeat_back_addr":"172.21.15.204:6824/13142","heartbeat_front_addr":"172.21.15.204:6821/13142","state":["exists","up"]},{"osd":6,"uuid":"09d4f546-1e94-43f6-a76a-98cb6b9fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6823","nonce":13144},{"type":"v1","addr":"172.21.15.204:6825","nonce":13144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13144},{"type":"v1","addr":"172.21.15.204:6827","nonce":13144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13144},{"type":"v1","addr":"172.21.15.204:6831","nonce":13144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13144},{"type":"v1","addr":"172.21.15.204:6829","nonce":13144}]},"public_addr":"172.21.15.204:6825/13144","cluster_addr":"172.21.15.204:6827/13144","heartbeat_back_addr":"172.21.15.204:6831/13144","heartbeat_front_addr":"172.21.15.204:6829/13144","state":["exists","up"]},{"osd":7,"uuid":"131feb5d-5cde-41b8-83bc-8a8258beed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13143},{"type":"v1","addr":"172.21.15.204:6809","nonce":13143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13143},{"type":"v1","addr":"172.21.15.204:6811","nonce":13143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13143},{"type":"v1","addr":"172.21.15.204:6816","nonce":13143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13143},{"type":"v1","addr":"172.21.15.204:6813","nonce":13143}]},"public_addr":"172.21.15.204:6809/13143","cluster_addr":"172.21.15.204:6811/13143","heartbeat_back_addr":"172.21.15.204:6816/13143","heartbeat_front_addr":"172.21.15.204:6813/13143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.968011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.963899+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.947850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.901012+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.943555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.938685+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.908440+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.930129+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-17T20:46:03.481 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-10-17T20:46:01.367245+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-10-17T20:46:03.482 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:03.483 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-10-17T20:46:03.754 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 8 2019-10-17T20:46:03.773 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-10-17T20:46:03.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-10-17T20:46:03.774 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:03.774 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-10-17T20:46:04.059 INFO:teuthology.orchestra.run.smithi092.stdout: 2019-10-17T20:46:04.072 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":3,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":12943},{"type":"v1","addr":"172.21.15.92:6801","nonce":12943}]},"active_addr":"172.21.15.92:6801/12943","active_change":"2019-10-17T20:45:50.058203+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"float","level":"advanced","flags":1,"default_value":"0.01","min":"0","max":"1","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the ratio between the fullest and least-full OSD is below this value then we stop trying to optimize placement.","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-10-17T20:46:04.075 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-10-17T20:46:04.882 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:04.882 INFO:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-10-17T20:46:05.157 INFO:teuthology.misc.health.smithi092.stdout: 2019-10-17T20:46:05.157 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":17,"fsid":"b8a78ffc-fd6b-49fb-a36f-703403180d29","created":"2019-10-17T20:45:47.750284+0000","modified":"2019-10-17T20:46:04.136340+0000","last_up_change":"2019-10-17T20:45:55.824589+0000","last_in_change":"2019-10-17T20:45:55.824589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-17T20:46:01.367245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8ab85492-12c6-4c7c-8201-f7659d0d84ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13089},{"type":"v1","addr":"172.21.15.92:6819","nonce":13089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":13089},{"type":"v1","addr":"172.21.15.92:6821","nonce":13089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6827","nonce":13089},{"type":"v1","addr":"172.21.15.92:6828","nonce":13089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13089},{"type":"v1","addr":"172.21.15.92:6825","nonce":13089}]},"public_addr":"172.21.15.92:6819/13089","cluster_addr":"172.21.15.92:6821/13089","heartbeat_back_addr":"172.21.15.92:6828/13089","heartbeat_front_addr":"172.21.15.92:6825/13089","state":["exists","up"]},{"osd":1,"uuid":"be05be4d-57ae-43bf-ac11-3341e5cabe9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":13091},{"type":"v1","addr":"172.21.15.92:6811","nonce":13091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13091},{"type":"v1","addr":"172.21.15.92:6813","nonce":13091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13091},{"type":"v1","addr":"172.21.15.92:6818","nonce":13091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":13091},{"type":"v1","addr":"172.21.15.92:6815","nonce":13091}]},"public_addr":"172.21.15.92:6811/13091","cluster_addr":"172.21.15.92:6813/13091","heartbeat_back_addr":"172.21.15.92:6818/13091","heartbeat_front_addr":"172.21.15.92:6815/13091","state":["exists","up"]},{"osd":2,"uuid":"9cbc29b0-9860-4f41-a3bb-aafecd59daba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13092},{"type":"v1","addr":"172.21.15.92:6824","nonce":13092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":13092},{"type":"v1","addr":"172.21.15.92:6829","nonce":13092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":13092},{"type":"v1","addr":"172.21.15.92:6833","nonce":13092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13092},{"type":"v1","addr":"172.21.15.92:6831","nonce":13092}]},"public_addr":"172.21.15.92:6824/13092","cluster_addr":"172.21.15.92:6829/13092","heartbeat_back_addr":"172.21.15.92:6833/13092","heartbeat_front_addr":"172.21.15.92:6831/13092","state":["exists","up"]},{"osd":3,"uuid":"110aa2aa-d522-4c91-aa4f-76ab73f79d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13090},{"type":"v1","addr":"172.21.15.92:6803","nonce":13090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13090},{"type":"v1","addr":"172.21.15.92:6805","nonce":13090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13090},{"type":"v1","addr":"172.21.15.92:6809","nonce":13090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13090},{"type":"v1","addr":"172.21.15.92:6807","nonce":13090}]},"public_addr":"172.21.15.92:6803/13090","cluster_addr":"172.21.15.92:6805/13090","heartbeat_back_addr":"172.21.15.92:6809/13090","heartbeat_front_addr":"172.21.15.92:6807/13090","state":["exists","up"]},{"osd":4,"uuid":"90b71d59-60f5-49b5-947c-fb012318f4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":13141},{"type":"v1","addr":"172.21.15.204:6801","nonce":13141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13141},{"type":"v1","addr":"172.21.15.204:6803","nonce":13141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13141},{"type":"v1","addr":"172.21.15.204:6807","nonce":13141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13141},{"type":"v1","addr":"172.21.15.204:6805","nonce":13141}]},"public_addr":"172.21.15.204:6801/13141","cluster_addr":"172.21.15.204:6803/13141","heartbeat_back_addr":"172.21.15.204:6807/13141","heartbeat_front_addr":"172.21.15.204:6805/13141","state":["exists","up"]},{"osd":5,"uuid":"7b3c3b9e-f75e-49e8-b195-ed56a924a36d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":13142},{"type":"v1","addr":"172.21.15.204:6817","nonce":13142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13142},{"type":"v1","addr":"172.21.15.204:6819","nonce":13142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13142},{"type":"v1","addr":"172.21.15.204:6824","nonce":13142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13142},{"type":"v1","addr":"172.21.15.204:6821","nonce":13142}]},"public_addr":"172.21.15.204:6817/13142","cluster_addr":"172.21.15.204:6819/13142","heartbeat_back_addr":"172.21.15.204:6824/13142","heartbeat_front_addr":"172.21.15.204:6821/13142","state":["exists","up"]},{"osd":6,"uuid":"09d4f546-1e94-43f6-a76a-98cb6b9fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6823","nonce":13144},{"type":"v1","addr":"172.21.15.204:6825","nonce":13144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13144},{"type":"v1","addr":"172.21.15.204:6827","nonce":13144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13144},{"type":"v1","addr":"172.21.15.204:6831","nonce":13144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13144},{"type":"v1","addr":"172.21.15.204:6829","nonce":13144}]},"public_addr":"172.21.15.204:6825/13144","cluster_addr":"172.21.15.204:6827/13144","heartbeat_back_addr":"172.21.15.204:6831/13144","heartbeat_front_addr":"172.21.15.204:6829/13144","state":["exists","up"]},{"osd":7,"uuid":"131feb5d-5cde-41b8-83bc-8a8258beed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13143},{"type":"v1","addr":"172.21.15.204:6809","nonce":13143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13143},{"type":"v1","addr":"172.21.15.204:6811","nonce":13143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13143},{"type":"v1","addr":"172.21.15.204:6816","nonce":13143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13143},{"type":"v1","addr":"172.21.15.204:6813","nonce":13143}]},"public_addr":"172.21.15.204:6809/13143","cluster_addr":"172.21.15.204:6811/13143","heartbeat_back_addr":"172.21.15.204:6816/13143","heartbeat_front_addr":"172.21.15.204:6813/13143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.968011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.963899+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.947850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.901012+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.943555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.938685+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.908440+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.930129+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-17T20:46:05.169 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-10-17T20:46:05.169 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:05.170 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-10-17T20:46:05.439 INFO:teuthology.orchestra.run.smithi092.stdout: 2019-10-17T20:46:05.439 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":17,"fsid":"b8a78ffc-fd6b-49fb-a36f-703403180d29","created":"2019-10-17T20:45:47.750284+0000","modified":"2019-10-17T20:46:04.136340+0000","last_up_change":"2019-10-17T20:45:55.824589+0000","last_in_change":"2019-10-17T20:45:55.824589+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-17T20:46:01.367245+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"8ab85492-12c6-4c7c-8201-f7659d0d84ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":13089},{"type":"v1","addr":"172.21.15.92:6819","nonce":13089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":13089},{"type":"v1","addr":"172.21.15.92:6821","nonce":13089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6827","nonce":13089},{"type":"v1","addr":"172.21.15.92:6828","nonce":13089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":13089},{"type":"v1","addr":"172.21.15.92:6825","nonce":13089}]},"public_addr":"172.21.15.92:6819/13089","cluster_addr":"172.21.15.92:6821/13089","heartbeat_back_addr":"172.21.15.92:6828/13089","heartbeat_front_addr":"172.21.15.92:6825/13089","state":["exists","up"]},{"osd":1,"uuid":"be05be4d-57ae-43bf-ac11-3341e5cabe9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":13091},{"type":"v1","addr":"172.21.15.92:6811","nonce":13091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":13091},{"type":"v1","addr":"172.21.15.92:6813","nonce":13091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":13091},{"type":"v1","addr":"172.21.15.92:6818","nonce":13091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":13091},{"type":"v1","addr":"172.21.15.92:6815","nonce":13091}]},"public_addr":"172.21.15.92:6811/13091","cluster_addr":"172.21.15.92:6813/13091","heartbeat_back_addr":"172.21.15.92:6818/13091","heartbeat_front_addr":"172.21.15.92:6815/13091","state":["exists","up"]},{"osd":2,"uuid":"9cbc29b0-9860-4f41-a3bb-aafecd59daba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":13092},{"type":"v1","addr":"172.21.15.92:6824","nonce":13092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":13092},{"type":"v1","addr":"172.21.15.92:6829","nonce":13092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":13092},{"type":"v1","addr":"172.21.15.92:6833","nonce":13092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":13092},{"type":"v1","addr":"172.21.15.92:6831","nonce":13092}]},"public_addr":"172.21.15.92:6824/13092","cluster_addr":"172.21.15.92:6829/13092","heartbeat_back_addr":"172.21.15.92:6833/13092","heartbeat_front_addr":"172.21.15.92:6831/13092","state":["exists","up"]},{"osd":3,"uuid":"110aa2aa-d522-4c91-aa4f-76ab73f79d7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":13090},{"type":"v1","addr":"172.21.15.92:6803","nonce":13090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":13090},{"type":"v1","addr":"172.21.15.92:6805","nonce":13090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":13090},{"type":"v1","addr":"172.21.15.92:6809","nonce":13090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":13090},{"type":"v1","addr":"172.21.15.92:6807","nonce":13090}]},"public_addr":"172.21.15.92:6803/13090","cluster_addr":"172.21.15.92:6805/13090","heartbeat_back_addr":"172.21.15.92:6809/13090","heartbeat_front_addr":"172.21.15.92:6807/13090","state":["exists","up"]},{"osd":4,"uuid":"90b71d59-60f5-49b5-947c-fb012318f4eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":13141},{"type":"v1","addr":"172.21.15.204:6801","nonce":13141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":13141},{"type":"v1","addr":"172.21.15.204:6803","nonce":13141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":13141},{"type":"v1","addr":"172.21.15.204:6807","nonce":13141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":13141},{"type":"v1","addr":"172.21.15.204:6805","nonce":13141}]},"public_addr":"172.21.15.204:6801/13141","cluster_addr":"172.21.15.204:6803/13141","heartbeat_back_addr":"172.21.15.204:6807/13141","heartbeat_front_addr":"172.21.15.204:6805/13141","state":["exists","up"]},{"osd":5,"uuid":"7b3c3b9e-f75e-49e8-b195-ed56a924a36d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":13142},{"type":"v1","addr":"172.21.15.204:6817","nonce":13142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":13142},{"type":"v1","addr":"172.21.15.204:6819","nonce":13142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":13142},{"type":"v1","addr":"172.21.15.204:6824","nonce":13142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":13142},{"type":"v1","addr":"172.21.15.204:6821","nonce":13142}]},"public_addr":"172.21.15.204:6817/13142","cluster_addr":"172.21.15.204:6819/13142","heartbeat_back_addr":"172.21.15.204:6824/13142","heartbeat_front_addr":"172.21.15.204:6821/13142","state":["exists","up"]},{"osd":6,"uuid":"09d4f546-1e94-43f6-a76a-98cb6b9fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6823","nonce":13144},{"type":"v1","addr":"172.21.15.204:6825","nonce":13144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":13144},{"type":"v1","addr":"172.21.15.204:6827","nonce":13144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":13144},{"type":"v1","addr":"172.21.15.204:6831","nonce":13144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":13144},{"type":"v1","addr":"172.21.15.204:6829","nonce":13144}]},"public_addr":"172.21.15.204:6825/13144","cluster_addr":"172.21.15.204:6827/13144","heartbeat_back_addr":"172.21.15.204:6831/13144","heartbeat_front_addr":"172.21.15.204:6829/13144","state":["exists","up"]},{"osd":7,"uuid":"131feb5d-5cde-41b8-83bc-8a8258beed6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":13143},{"type":"v1","addr":"172.21.15.204:6809","nonce":13143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":13143},{"type":"v1","addr":"172.21.15.204:6811","nonce":13143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":13143},{"type":"v1","addr":"172.21.15.204:6816","nonce":13143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":13143},{"type":"v1","addr":"172.21.15.204:6813","nonce":13143}]},"public_addr":"172.21.15.204:6809/13143","cluster_addr":"172.21.15.204:6811/13143","heartbeat_back_addr":"172.21.15.204:6816/13143","heartbeat_front_addr":"172.21.15.204:6813/13143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.968011+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.963899+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.947850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.901012+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.943555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.938685+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.908440+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-10-17T20:45:53.930129+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-10-17T20:46:05.452 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:05.452 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-10-17T20:46:05.608 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:05.609 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:05.609 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-10-17T20:46:05.761 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:05.761 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:05.761 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-10-17T20:46:05.913 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:05.913 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:05.913 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-10-17T20:46:06.075 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:06.076 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:06.076 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-10-17T20:46:06.236 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:06.236 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:06.236 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-10-17T20:46:06.387 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:06.388 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:06.388 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-10-17T20:46:06.541 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:06.541 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:06.541 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-10-17T20:46:06.686 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:06.687 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:06.687 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-10-17T20:46:06.962 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:06.974 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2019-10-17T20:46:06.975 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:06.975 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-10-17T20:46:07.257 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:07.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2019-10-17T20:46:07.269 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:07.269 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-10-17T20:46:07.540 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:07.553 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2019-10-17T20:46:07.553 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:07.554 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-10-17T20:46:07.821 INFO:teuthology.orchestra.run.smithi092.stdout:55834574853 2019-10-17T20:46:07.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2019-10-17T20:46:07.834 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:07.834 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-10-17T20:46:08.119 INFO:teuthology.orchestra.run.smithi092.stdout:55834574852 2019-10-17T20:46:08.132 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2019-10-17T20:46:09.133 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:09.133 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-10-17T20:46:09.412 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2019-10-17T20:46:09.423 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.4 2019-10-17T20:46:09.423 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:09.423 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-10-17T20:46:09.690 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2019-10-17T20:46:09.707 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2019-10-17T20:46:09.708 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:09.708 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-10-17T20:46:09.994 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2019-10-17T20:46:10.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.6 2019-10-17T20:46:10.006 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:10.006 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-10-17T20:46:10.323 INFO:teuthology.orchestra.run.smithi092.stdout:55834574854 2019-10-17T20:46:10.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2019-10-17T20:46:10.335 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-10-17T20:46:10.335 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:10.335 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-17T20:46:10.625 INFO:teuthology.orchestra.run.smithi092.stdout: 2019-10-17T20:46:10.626 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2019-10-17T20:46:10.639 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2019-10-17T20:46:10.067967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887424,"kb_used_data":747707968,"kb_used_omap":334,"kb_used_meta":0,"kb_avail":747707616,"statfs":{"total":767585320960,"available":765652598784,"internally_reserved":360448,"allocated":765652959232,"data_stored":765652959232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":342829,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.152491+0000","last_change":"2019-10-17T20:46:03.150863+0000","last_active":"2019-10-17T20:46:03.152491+0000","last_peered":"2019-10-17T20:46:03.152491+0000","last_clean":"2019-10-17T20:46:03.152491+0000","last_became_active":"2019-10-17T20:46:03.150466+0000","last_became_peered":"2019-10-17T20:46:03.150466+0000","last_unstale":"2019-10-17T20:46:03.152491+0000","last_undegraded":"2019-10-17T20:46:03.152491+0000","last_fullsized":"2019-10-17T20:46:03.152491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149821+0000","last_change":"2019-10-17T20:46:03.147896+0000","last_active":"2019-10-17T20:46:03.149821+0000","last_peered":"2019-10-17T20:46:03.149821+0000","last_clean":"2019-10-17T20:46:03.149821+0000","last_became_active":"2019-10-17T20:46:03.147270+0000","last_became_peered":"2019-10-17T20:46:03.147270+0000","last_unstale":"2019-10-17T20:46:03.149821+0000","last_undegraded":"2019-10-17T20:46:03.149821+0000","last_fullsized":"2019-10-17T20:46:03.149821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149921+0000","last_change":"2019-10-17T20:46:03.148117+0000","last_active":"2019-10-17T20:46:03.149921+0000","last_peered":"2019-10-17T20:46:03.149921+0000","last_clean":"2019-10-17T20:46:03.149921+0000","last_became_active":"2019-10-17T20:46:03.147385+0000","last_became_peered":"2019-10-17T20:46:03.147385+0000","last_unstale":"2019-10-17T20:46:03.149921+0000","last_undegraded":"2019-10-17T20:46:03.149921+0000","last_fullsized":"2019-10-17T20:46:03.149921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149814+0000","last_change":"2019-10-17T20:46:03.148414+0000","last_active":"2019-10-17T20:46:03.149814+0000","last_peered":"2019-10-17T20:46:03.149814+0000","last_clean":"2019-10-17T20:46:03.149814+0000","last_became_active":"2019-10-17T20:46:03.148057+0000","last_became_peered":"2019-10-17T20:46:03.148057+0000","last_unstale":"2019-10-17T20:46:03.149814+0000","last_undegraded":"2019-10-17T20:46:03.149814+0000","last_fullsized":"2019-10-17T20:46:03.149814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149828+0000","last_change":"2019-10-17T20:46:03.148432+0000","last_active":"2019-10-17T20:46:03.149828+0000","last_peered":"2019-10-17T20:46:03.149828+0000","last_clean":"2019-10-17T20:46:03.149828+0000","last_became_active":"2019-10-17T20:46:03.148066+0000","last_became_peered":"2019-10-17T20:46:03.148066+0000","last_unstale":"2019-10-17T20:46:03.149828+0000","last_undegraded":"2019-10-17T20:46:03.149828+0000","last_fullsized":"2019-10-17T20:46:03.149828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.147482+0000","last_change":"2019-10-17T20:46:03.145849+0000","last_active":"2019-10-17T20:46:03.147482+0000","last_peered":"2019-10-17T20:46:03.147482+0000","last_clean":"2019-10-17T20:46:03.147482+0000","last_became_active":"2019-10-17T20:46:03.145511+0000","last_became_peered":"2019-10-17T20:46:03.145511+0000","last_unstale":"2019-10-17T20:46:03.147482+0000","last_undegraded":"2019-10-17T20:46:03.147482+0000","last_fullsized":"2019-10-17T20:46:03.147482+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.147346+0000","last_change":"2019-10-17T20:46:03.144786+0000","last_active":"2019-10-17T20:46:03.147346+0000","last_peered":"2019-10-17T20:46:03.147346+0000","last_clean":"2019-10-17T20:46:03.147346+0000","last_became_active":"2019-10-17T20:46:03.144480+0000","last_became_peered":"2019-10-17T20:46:03.144480+0000","last_unstale":"2019-10-17T20:46:03.147346+0000","last_undegraded":"2019-10-17T20:46:03.147346+0000","last_fullsized":"2019-10-17T20:46:03.147346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149729+0000","last_change":"2019-10-17T20:46:03.147706+0000","last_active":"2019-10-17T20:46:03.149729+0000","last_peered":"2019-10-17T20:46:03.149729+0000","last_clean":"2019-10-17T20:46:03.149729+0000","last_became_active":"2019-10-17T20:46:03.147402+0000","last_became_peered":"2019-10-17T20:46:03.147402+0000","last_unstale":"2019-10-17T20:46:03.149729+0000","last_undegraded":"2019-10-17T20:46:03.149729+0000","last_fullsized":"2019-10-17T20:46:03.149729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235912,"kb_used_data":93463496,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":93463468,"statfs":{"total":95948165120,"available":95706591232,"internally_reserved":28672,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.981}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.800}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.405}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.578}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.921}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.344}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.303}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235964,"kb_used_data":93463496,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":93463416,"statfs":{"total":95948165120,"available":95706537984,"internally_reserved":81920,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52440,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.108}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.917}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.856}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.076}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.186}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.365}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235940,"kb_used_data":93463496,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":93463440,"statfs":{"total":95948165120,"available":95706562560,"internally_reserved":57344,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47714,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.697}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.570}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.923}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.239}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.345}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.045}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235928,"kb_used_data":93463496,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":93463452,"statfs":{"total":95948165120,"available":95706574848,"internally_reserved":45056,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42837,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.550}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.841}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.583}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.688}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.024}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.774}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235912,"kb_used_data":93463496,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":93463468,"statfs":{"total":95948165120,"available":95706591232,"internally_reserved":28672,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38042,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.820}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.713}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.884}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.949}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.776}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.157}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.009}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235924,"kb_used_data":93463496,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":93463456,"statfs":{"total":95948165120,"available":95706578944,"internally_reserved":40960,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42900,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.532}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.649}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.741}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.086}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.685}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235928,"kb_used_data":93463496,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":93463452,"statfs":{"total":95948165120,"available":95706574848,"internally_reserved":45056,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42856,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.577}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.681}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.922}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.767}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.104}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.464}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235916,"kb_used_data":93463496,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":93463464,"statfs":{"total":95948165120,"available":95706587136,"internally_reserved":32768,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37998,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.078}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.326}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.157}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.845}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.646}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.215}]}]}]}} 2019-10-17T20:46:10.641 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:10.641 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-10-17T20:46:10.922 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2019-10-17T20:46:10.923 INFO:teuthology.orchestra.run.smithi092.stdout: 2019-10-17T20:46:10.934 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2019-10-17T20:46:10.067967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887424,"kb_used_data":747707968,"kb_used_omap":334,"kb_used_meta":0,"kb_avail":747707616,"statfs":{"total":767585320960,"available":765652598784,"internally_reserved":360448,"allocated":765652959232,"data_stored":765652959232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":342829,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.152491+0000","last_change":"2019-10-17T20:46:03.150863+0000","last_active":"2019-10-17T20:46:03.152491+0000","last_peered":"2019-10-17T20:46:03.152491+0000","last_clean":"2019-10-17T20:46:03.152491+0000","last_became_active":"2019-10-17T20:46:03.150466+0000","last_became_peered":"2019-10-17T20:46:03.150466+0000","last_unstale":"2019-10-17T20:46:03.152491+0000","last_undegraded":"2019-10-17T20:46:03.152491+0000","last_fullsized":"2019-10-17T20:46:03.152491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149821+0000","last_change":"2019-10-17T20:46:03.147896+0000","last_active":"2019-10-17T20:46:03.149821+0000","last_peered":"2019-10-17T20:46:03.149821+0000","last_clean":"2019-10-17T20:46:03.149821+0000","last_became_active":"2019-10-17T20:46:03.147270+0000","last_became_peered":"2019-10-17T20:46:03.147270+0000","last_unstale":"2019-10-17T20:46:03.149821+0000","last_undegraded":"2019-10-17T20:46:03.149821+0000","last_fullsized":"2019-10-17T20:46:03.149821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149921+0000","last_change":"2019-10-17T20:46:03.148117+0000","last_active":"2019-10-17T20:46:03.149921+0000","last_peered":"2019-10-17T20:46:03.149921+0000","last_clean":"2019-10-17T20:46:03.149921+0000","last_became_active":"2019-10-17T20:46:03.147385+0000","last_became_peered":"2019-10-17T20:46:03.147385+0000","last_unstale":"2019-10-17T20:46:03.149921+0000","last_undegraded":"2019-10-17T20:46:03.149921+0000","last_fullsized":"2019-10-17T20:46:03.149921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149814+0000","last_change":"2019-10-17T20:46:03.148414+0000","last_active":"2019-10-17T20:46:03.149814+0000","last_peered":"2019-10-17T20:46:03.149814+0000","last_clean":"2019-10-17T20:46:03.149814+0000","last_became_active":"2019-10-17T20:46:03.148057+0000","last_became_peered":"2019-10-17T20:46:03.148057+0000","last_unstale":"2019-10-17T20:46:03.149814+0000","last_undegraded":"2019-10-17T20:46:03.149814+0000","last_fullsized":"2019-10-17T20:46:03.149814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149828+0000","last_change":"2019-10-17T20:46:03.148432+0000","last_active":"2019-10-17T20:46:03.149828+0000","last_peered":"2019-10-17T20:46:03.149828+0000","last_clean":"2019-10-17T20:46:03.149828+0000","last_became_active":"2019-10-17T20:46:03.148066+0000","last_became_peered":"2019-10-17T20:46:03.148066+0000","last_unstale":"2019-10-17T20:46:03.149828+0000","last_undegraded":"2019-10-17T20:46:03.149828+0000","last_fullsized":"2019-10-17T20:46:03.149828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.147482+0000","last_change":"2019-10-17T20:46:03.145849+0000","last_active":"2019-10-17T20:46:03.147482+0000","last_peered":"2019-10-17T20:46:03.147482+0000","last_clean":"2019-10-17T20:46:03.147482+0000","last_became_active":"2019-10-17T20:46:03.145511+0000","last_became_peered":"2019-10-17T20:46:03.145511+0000","last_unstale":"2019-10-17T20:46:03.147482+0000","last_undegraded":"2019-10-17T20:46:03.147482+0000","last_fullsized":"2019-10-17T20:46:03.147482+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.147346+0000","last_change":"2019-10-17T20:46:03.144786+0000","last_active":"2019-10-17T20:46:03.147346+0000","last_peered":"2019-10-17T20:46:03.147346+0000","last_clean":"2019-10-17T20:46:03.147346+0000","last_became_active":"2019-10-17T20:46:03.144480+0000","last_became_peered":"2019-10-17T20:46:03.144480+0000","last_unstale":"2019-10-17T20:46:03.147346+0000","last_undegraded":"2019-10-17T20:46:03.147346+0000","last_fullsized":"2019-10-17T20:46:03.147346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"16","state":"active+clean","last_fresh":"2019-10-17T20:46:03.149729+0000","last_change":"2019-10-17T20:46:03.147706+0000","last_active":"2019-10-17T20:46:03.149729+0000","last_peered":"2019-10-17T20:46:03.149729+0000","last_clean":"2019-10-17T20:46:03.149729+0000","last_became_active":"2019-10-17T20:46:03.147402+0000","last_became_peered":"2019-10-17T20:46:03.147402+0000","last_unstale":"2019-10-17T20:46:03.149729+0000","last_undegraded":"2019-10-17T20:46:03.149729+0000","last_fullsized":"2019-10-17T20:46:03.149729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-10-17T20:46:02.123712+0000","last_clean_scrub_stamp":"2019-10-17T20:46:02.123712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235912,"kb_used_data":93463496,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":93463468,"statfs":{"total":95948165120,"available":95706591232,"internally_reserved":28672,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.981}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.800}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.405}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.578}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.921}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.344}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.303}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235964,"kb_used_data":93463496,"kb_used_omap":51,"kb_used_meta":0,"kb_avail":93463416,"statfs":{"total":95948165120,"available":95706537984,"internally_reserved":81920,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52440,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.108}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.990}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.917}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.856}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.076}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.186}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.365}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235940,"kb_used_data":93463496,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":93463440,"statfs":{"total":95948165120,"available":95706562560,"internally_reserved":57344,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47714,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.697}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.570}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.923}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.239}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.345}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.045}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235928,"kb_used_data":93463496,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":93463452,"statfs":{"total":95948165120,"available":95706574848,"internally_reserved":45056,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42837,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.550}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.841}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.583}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.688}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.024}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.774}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235912,"kb_used_data":93463496,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":93463468,"statfs":{"total":95948165120,"available":95706591232,"internally_reserved":28672,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38042,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.820}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.713}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.884}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.949}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.776}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.157}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.009}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235924,"kb_used_data":93463496,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":93463456,"statfs":{"total":95948165120,"available":95706578944,"internally_reserved":40960,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42900,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.532}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.649}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.741}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.086}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.685}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235928,"kb_used_data":93463496,"kb_used_omap":41,"kb_used_meta":0,"kb_avail":93463452,"statfs":{"total":95948165120,"available":95706574848,"internally_reserved":45056,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42856,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.577}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.681}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.922}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.767}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.391}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.104}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.464}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235916,"kb_used_data":93463496,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":93463464,"statfs":{"total":95948165120,"available":95706587136,"internally_reserved":32768,"allocated":95706619904,"data_stored":95706619904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37998,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.078}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.326}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.157}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.014}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.845}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.646}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.215}]}]}]}} 2019-10-17T20:46:10.937 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-10-17T20:46:10.937 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-10-17T20:46:10.937 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:10.937 INFO:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2019-10-17T20:46:11.234 INFO:teuthology.misc.health.smithi092.stdout:HEALTH_OK 2019-10-17T20:46:11.245 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2019-10-17T20:46:11.245 INFO:teuthology.run_tasks:Running task rados... 2019-10-17T20:46:11.257 INFO:tasks.rados:Beginning rados... 2019-10-17T20:46:11.257 DEBUG:teuthology.run_tasks:Unwinding manager rados 2019-10-17T20:46:11.269 INFO:tasks.rados:joining rados 2019-10-17T20:46:11.269 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2019-10-17T20:46:11.269 INFO:tasks.rados:starting run 0 out of 1 2019-10-17T20:46:11.269 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2019-10-17T20:46:11.269 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:11.270 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2019-10-17T20:46:12.561 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_0' created 2019-10-17T20:46:12.578 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:12.578 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2019-10-17T20:46:13.580 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rados' on pool 'unique_pool_0' 2019-10-17T20:46:14.599 INFO:teuthology.orchestra.run.smithi092:Running: 2019-10-17T20:46:14.599 INFO:teuthology.orchestra.run.smithi092:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op write_excl 22 --op delete 10 --pool unique_pool_0 2019-10-17T20:46:14.618 INFO:tasks.rados.rados.0.smithi092.stdout:adding op weight read -> 45 2019-10-17T20:46:14.618 INFO:tasks.rados.rados.0.smithi092.stdout:adding op weight write -> 22 2019-10-17T20:46:14.618 INFO:tasks.rados.rados.0.smithi092.stdout:adding op weight write_excl -> 22 2019-10-17T20:46:14.618 INFO:tasks.rados.rados.0.smithi092.stdout:adding op weight delete -> 10 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout:ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout:Configuration: 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout: Number of operations: 4000 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout: Number of objects: 500 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout: Max in flight operations: 16 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout: Object size (in bytes): 4000000 2019-10-17T20:46:14.619 INFO:tasks.rados.rados.0.smithi092.stdout: Write stride min: 400000 2019-10-17T20:46:14.620 INFO:tasks.rados.rados.0.smithi092.stdout: Write stride max: 800000 2019-10-17T20:46:15.562 INFO:tasks.rados.rados.0.smithi092.stdout:1: write initial oid 1 2019-10-17T20:46:15.562 INFO:tasks.rados.rados.0.smithi092.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2019-10-17T20:46:15.571 INFO:tasks.rados.rados.0.smithi092.stdout:1: writing smithi09214413-1 from 416807 to 892056 tid 1 2019-10-17T20:46:15.581 INFO:tasks.rados.rados.0.smithi092.stdout:1: writing smithi09214413-1 from 1542129 to 2016807 tid 2 2019-10-17T20:46:15.581 INFO:tasks.rados.rados.0.smithi092.stdout:2: write initial oid 2 2019-10-17T20:46:15.581 INFO:tasks.rados.rados.0.smithi092.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2019-10-17T20:46:15.586 INFO:tasks.rados.rados.0.smithi092.stdout:2: writing smithi09214413-2 from 416807 to 892056 tid 1 2019-10-17T20:46:15.593 INFO:tasks.rados.rados.0.smithi092.stdout:2: writing smithi09214413-2 from 1542129 to 2016807 tid 2 2019-10-17T20:46:15.593 INFO:tasks.rados.rados.0.smithi092.stdout:3: write initial oid 3 2019-10-17T20:46:15.593 INFO:tasks.rados.rados.0.smithi092.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2019-10-17T20:46:15.599 INFO:tasks.rados.rados.0.smithi092.stdout:3: writing smithi09214413-3 from 433614 to 984112 tid 1 2019-10-17T20:46:15.605 INFO:tasks.rados.rados.0.smithi092.stdout:3: writing smithi09214413-3 from 1600611 to 2033614 tid 2 2019-10-17T20:46:15.605 INFO:tasks.rados.rados.0.smithi092.stdout:4: write initial oid 4 2019-10-17T20:46:15.605 INFO:tasks.rados.rados.0.smithi092.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2019-10-17T20:46:15.611 INFO:tasks.rados.rados.0.smithi092.stdout:4: writing smithi09214413-4 from 450421 to 1076168 tid 1 2019-10-17T20:46:15.617 INFO:tasks.rados.rados.0.smithi092.stdout:4: writing smithi09214413-4 from 1659093 to 2050421 tid 2 2019-10-17T20:46:15.617 INFO:tasks.rados.rados.0.smithi092.stdout:5: write initial oid 5 2019-10-17T20:46:15.617 INFO:tasks.rados.rados.0.smithi092.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2019-10-17T20:46:15.623 INFO:tasks.rados.rados.0.smithi092.stdout:5: writing smithi09214413-5 from 467228 to 1168224 tid 1 2019-10-17T20:46:15.629 INFO:tasks.rados.rados.0.smithi092.stdout:5: writing smithi09214413-5 from 1717575 to 2067228 tid 2 2019-10-17T20:46:15.629 INFO:tasks.rados.rados.0.smithi092.stdout:6: write initial oid 6 2019-10-17T20:46:15.629 INFO:tasks.rados.rados.0.smithi092.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2019-10-17T20:46:15.636 INFO:tasks.rados.rados.0.smithi092.stdout:6: writing smithi09214413-6 from 484035 to 1260280 tid 1 2019-10-17T20:46:15.641 INFO:tasks.rados.rados.0.smithi092.stdout:6: writing smithi09214413-6 from 2059704 to 2084035 tid 2 2019-10-17T20:46:15.641 INFO:tasks.rados.rados.0.smithi092.stdout:7: write initial oid 7 2019-10-17T20:46:15.642 INFO:tasks.rados.rados.0.smithi092.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2019-10-17T20:46:15.647 INFO:tasks.rados.rados.0.smithi092.stdout:7: writing smithi09214413-7 from 500842 to 952336 tid 1 2019-10-17T20:46:15.654 INFO:tasks.rados.rados.0.smithi092.stdout:7: writing smithi09214413-7 from 1718186 to 2100842 tid 2 2019-10-17T20:46:15.655 INFO:tasks.rados.rados.0.smithi092.stdout:8: write initial oid 8 2019-10-17T20:46:15.655 INFO:tasks.rados.rados.0.smithi092.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2019-10-17T20:46:15.661 INFO:tasks.rados.rados.0.smithi092.stdout:8: writing smithi09214413-8 from 517649 to 1044392 tid 1 2019-10-17T20:46:15.668 INFO:tasks.rados.rados.0.smithi092.stdout:8: writing smithi09214413-8 from 1776668 to 2117649 tid 2 2019-10-17T20:46:15.668 INFO:tasks.rados.rados.0.smithi092.stdout:9: write initial oid 9 2019-10-17T20:46:15.668 INFO:tasks.rados.rados.0.smithi092.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2019-10-17T20:46:15.676 INFO:tasks.rados.rados.0.smithi092.stdout:9: writing smithi09214413-9 from 534456 to 1252801 tid 1 2019-10-17T20:46:15.682 INFO:tasks.rados.rados.0.smithi092.stdout:9: writing smithi09214413-9 from 1951503 to 2134456 tid 2 2019-10-17T20:46:15.682 INFO:tasks.rados.rados.0.smithi092.stdout:10: write initial oid 10 2019-10-17T20:46:15.682 INFO:tasks.rados.rados.0.smithi092.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2019-10-17T20:46:15.690 INFO:tasks.rados.rados.0.smithi092.stdout:10: writing smithi09214413-10 from 551263 to 1344857 tid 1 2019-10-17T20:46:15.696 INFO:tasks.rados.rados.0.smithi092.stdout:10: writing smithi09214413-10 from 1893632 to 2151263 tid 2 2019-10-17T20:46:15.696 INFO:tasks.rados.rados.0.smithi092.stdout:11: write initial oid 11 2019-10-17T20:46:15.696 INFO:tasks.rados.rados.0.smithi092.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2019-10-17T20:46:15.702 INFO:tasks.rados.rados.0.smithi092.stdout:11: writing smithi09214413-11 from 568070 to 1036913 tid 1 2019-10-17T20:46:15.710 INFO:tasks.rados.rados.0.smithi092.stdout:11: writing smithi09214413-11 from 1552114 to 2168070 tid 2 2019-10-17T20:46:15.710 INFO:tasks.rados.rados.0.smithi092.stdout:12: write initial oid 12 2019-10-17T20:46:15.711 INFO:tasks.rados.rados.0.smithi092.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2019-10-17T20:46:15.717 INFO:tasks.rados.rados.0.smithi092.stdout:12: writing smithi09214413-12 from 584877 to 1128969 tid 1 2019-10-17T20:46:15.725 INFO:tasks.rados.rados.0.smithi092.stdout:12: writing smithi09214413-12 from 1610596 to 2172599 tid 2 2019-10-17T20:46:15.725 INFO:tasks.rados.rados.0.smithi092.stdout:13: write initial oid 13 2019-10-17T20:46:15.725 INFO:tasks.rados.rados.0.smithi092.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2019-10-17T20:46:15.732 INFO:tasks.rados.rados.0.smithi092.stdout:13: writing smithi09214413-13 from 601684 to 1221025 tid 1 2019-10-17T20:46:15.739 INFO:tasks.rados.rados.0.smithi092.stdout:13: writing smithi09214413-13 from 1669078 to 2201684 tid 2 2019-10-17T20:46:15.740 INFO:tasks.rados.rados.0.smithi092.stdout:14: write initial oid 14 2019-10-17T20:46:15.740 INFO:tasks.rados.rados.0.smithi092.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2019-10-17T20:46:15.748 INFO:tasks.rados.rados.0.smithi092.stdout:14: writing smithi09214413-14 from 618491 to 1313081 tid 1 2019-10-17T20:46:15.753 INFO:tasks.rados.rados.0.smithi092.stdout:14: writing smithi09214413-14 from 2011207 to 2218491 tid 2 2019-10-17T20:46:15.753 INFO:tasks.rados.rados.0.smithi092.stdout:15: write initial oid 15 2019-10-17T20:46:15.753 INFO:tasks.rados.rados.0.smithi092.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2019-10-17T20:46:15.762 INFO:tasks.rados.rados.0.smithi092.stdout:15: writing smithi09214413-15 from 635298 to 1405137 tid 1 2019-10-17T20:46:15.767 INFO:tasks.rados.rados.0.smithi092.stdout:15: writing smithi09214413-15 from 2069689 to 2235298 tid 2 2019-10-17T20:46:15.768 INFO:tasks.rados.rados.0.smithi092.stdout:16: write initial oid 16 2019-10-17T20:46:15.768 INFO:tasks.rados.rados.0.smithi092.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2019-10-17T20:46:15.774 INFO:tasks.rados.rados.0.smithi092.stdout:16: writing smithi09214413-16 from 652105 to 1097193 tid 1 2019-10-17T20:46:15.781 INFO:tasks.rados.rados.0.smithi092.stdout:16: writing smithi09214413-16 from 1728171 to 2181159 tid 2 2019-10-17T20:46:15.782 INFO:tasks.rados.rados.0.smithi092.stdout: waiting on 16 2019-10-17T20:46:52.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:52.128+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:53.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:53.124+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:53.997 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:53.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:46:54.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:54.120+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:55.006 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:55.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:46:55.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:55.144+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:55.958 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:55.956+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:46:56.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:56.164+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:56.192 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:46:56.188+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:46:56.974 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:56.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:46:57.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:57.152+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:57.190 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:46:57.184+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:46:57.976 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:57.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:46:58.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:58.144+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:58.176 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:46:58.172+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:46:58.990 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:58.988+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:46:59.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:46:59.132+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:46:59.220 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:46:59.216+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:46:59.997 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:46:59.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:00.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:00.108+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:00.219 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:00.212+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:00.969 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:00.964+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:01.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:01.088+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:01.262 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:01.256+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:02.000 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:01.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:02.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:02.096+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:02.220 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:02.216+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:03.021 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:03.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:03.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:03.128+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:03.221 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:03.216+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:04.070 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:04.068+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:04.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:04.096+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:04.260 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:04.256+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:05.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:05.088+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:05.113 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:05.108+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:05.237 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:05.232+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:06.074 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:06.072+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:06.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:06.096+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:06.225 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:06.220+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:07.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:07.072+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:07.083 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:07.080+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:07.191 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:07.184+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:08.038 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:08.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:08.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:08.068+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:08.179 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:08.176+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:08.999 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:08.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:09.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:09.076+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:09.212 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:09.204+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:10.017 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:10.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:10.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:10.080+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:10.168 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:10.164+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:10.987 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:10.984+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:11.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:11.056+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:11.174 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:11.168+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:12.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:12.008+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:12.025 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:12.024+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:12.133 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:12.128+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:12.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:12.968+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:13.048 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:13.044+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:13.124 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:13.120+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:14.005 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:14.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:14.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:14.004+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:14.080 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:14.076+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:15.017 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:15.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:15.040 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:15.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:15.044+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:16.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:16.004+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:16.013 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:16.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:16.066 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:16.064+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:16.983 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:16.976+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:17.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:17.012+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:17.071 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:17.068+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:17.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:17.992+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:18.008 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:18.004+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:18.074 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:18.072+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:18.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:18.968+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:18.988 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:18.984+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:19.062 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:19.060+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:19.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:19.940+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:19.973 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:19.968+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:20.012 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:20.008+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:20.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:20.900+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:20.967 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:20.964+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:21.007 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:21.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:21.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:21.936+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:21.998 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:21.992+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:22.015 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:22.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:22.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:22.936+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:23.031 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:23.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:23.038 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:23.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:23.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:23.916+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:23.995 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:23.988+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:24.020 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:24.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:24.892+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:24.972 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:24.968+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:25.006 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:25.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:25.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:25.896+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:25.963 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:25.960+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:26.007 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:25.996+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:26.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:26.900+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:26.988 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:26.984+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:27.044 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:27.040+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:27.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:27.860+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:28.031 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:28.028+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:28.045 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:28.040+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:28.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:28.840+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:29.022 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:29.020+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:29.031 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:29.024+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:29.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:29.860+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:29.998 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:29.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:30.075 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:30.072+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:30.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:30.856+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:30.968 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:30.964+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:31.069 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:31.064+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:31.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:31.864+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:31.995 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:31.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:32.063 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:32.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:32.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:32.820+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:33.038 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:33.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:33.109 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:33.104+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:33.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:33.804+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:33.996 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:33.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:34.068 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:34.064+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:34.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:34.804+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:35.019 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:35.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:35.024 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:35.020+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:35.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:35.844+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:35.980 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:35.976+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:36.045 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:36.040+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:36.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:36.852+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:37.030 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:37.028+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:37.088 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:37.088 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:37.088 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:37.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:37.828+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:38.001 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:38.000+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:38.042 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:38.042 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:38.042 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:38.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:38.828+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:38.993 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:38.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:39.011 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:39.011 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:39.012 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:39.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:39.840+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:39.973 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:39.974 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:39.974 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:39.974 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:39.974 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:40.049 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:40.049 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:40.049 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:40.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:40.864+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:40.951 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:40.952 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:40.952 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:40.952 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:40.952 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:41.003 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:41.003 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:41.003 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:41.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:41.844+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.137657+0000 front 2019-10-17T20:47:15.137527+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:41.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:41.844+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.138231+0000 front 2019-10-17T20:47:15.137482+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:41.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:41.844+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:41.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:41.844+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.138138+0000 front 2019-10-17T20:47:15.137685+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:41.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:41.844+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.137704+0000 front 2019-10-17T20:47:15.137507+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:41.909 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:41.909 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:41.909 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:41.909 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:41.909 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:42.031 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:42.032 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:42.032 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:42.032 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:47:42.032 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:47:42.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:42.816+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.137657+0000 front 2019-10-17T20:47:15.137527+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:42.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:42.816+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.138231+0000 front 2019-10-17T20:47:15.137482+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:42.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:42.816+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:42.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:42.816+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.138138+0000 front 2019-10-17T20:47:15.137685+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:42.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:42.816+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.137704+0000 front 2019-10-17T20:47:15.137507+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:42.874 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:42.874 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:42.874 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:42.874 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:42.874 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:43.064 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:43.064 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:43.064 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:43.065 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:47:43.065 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:47:43.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:43.768+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.137657+0000 front 2019-10-17T20:47:15.137527+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:43.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:43.768+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.138231+0000 front 2019-10-17T20:47:15.137482+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:43.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:43.768+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:27.934274+0000 front 2019-10-17T20:46:27.935057+0000 (oldest deadline 2019-10-17T20:46:52.033879+0000) 2019-10-17T20:47:43.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:43.768+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.138138+0000 front 2019-10-17T20:47:15.137685+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:43.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:47:43.768+0000 7fd46a48c700 -1 osd.1 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.137704+0000 front 2019-10-17T20:47:15.137507+0000 (oldest deadline 2019-10-17T20:47:41.037085+0000) 2019-10-17T20:47:43.824 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:43.824 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:43.825 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:47:43.825 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:43.825 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:47:44.014 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:44.015 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:47:44.015 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:47:44.015 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:47:44.015 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:47:44.012+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:48:23.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:23.848+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:23.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:23.848+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:24.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:24.840+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:24.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:24.840+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:25.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:25.852+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:25.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:25.852+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:26.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:26.868+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:26.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:26.868+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:27.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:27.904+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:27.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:27.904+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:28.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:28.928+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:28.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:28.928+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:29.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:29.944+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:29.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:29.944+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:30.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:30.924+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:30.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:30.924+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:31.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:31.944+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:31.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:31.944+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:32.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:32.920+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:32.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:32.920+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:33.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:33.916+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:33.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:33.916+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:34.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:34.900+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:34.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:34.900+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:35.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:35.940+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:35.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:35.940+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:36.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:36.908+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:36.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:36.908+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:37.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:37.884+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:37.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:37.884+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:38.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:38.856+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:38.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:38.856+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:39.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:39.888+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:39.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:39.888+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:40.876+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:40.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:40.876+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:41.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:41.892+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:41.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:41.892+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:42.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:42.908+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:42.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:42.908+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:43.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:43.940+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:43.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:43.940+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:44.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:44.976+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:44.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:44.976+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:45.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:45.936+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:45.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:45.936+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:46.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:46.904+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:46.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:46.904+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:47.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:47.952+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:47.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:47.952+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:48.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:48.988+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:48.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:48.988+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:49.263 INFO:tasks.ceph.osd.4.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-17T20:48:49.263 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7efdc46f0700 thread_name:tp_osd_tp 2019-10-17T20:48:49.267 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:48:49.268 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7efdf3e32890] 2019-10-17T20:48:49.268 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55a05769d16e] 2019-10-17T20:48:49.268 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55a0576fa20c] 2019-10-17T20:48:49.268 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55a0576fa89f] 2019-10-17T20:48:49.268 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55a0576ff598] 2019-10-17T20:48:49.268 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55a05770ab71] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55a0575913cb] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55a0577f6397] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55a0575acd3c] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55a057be58cc] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a057be8a40] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 12: (()+0x76db) [0x7efdf3e276db] 2019-10-17T20:48:49.269 INFO:tasks.ceph.osd.4.smithi204.stderr: 13: (clone()+0x3f) [0x7efdf2bc788f] 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr:2019-10-17T20:48:49.264+0000 7efdc46f0700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7efdc46f0700 thread_name:tp_osd_tp 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7efdf3e32890] 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55a05769d16e] 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55a0576fa20c] 2019-10-17T20:48:49.270 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55a0576fa89f] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55a0576ff598] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55a05770ab71] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55a0575913cb] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55a0577f6397] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55a0575acd3c] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55a057be58cc] 2019-10-17T20:48:49.271 INFO:tasks.ceph.osd.4.smithi204.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a057be8a40] 2019-10-17T20:48:49.272 INFO:tasks.ceph.osd.4.smithi204.stderr: 12: (()+0x76db) [0x7efdf3e276db] 2019-10-17T20:48:49.272 INFO:tasks.ceph.osd.4.smithi204.stderr: 13: (clone()+0x3f) [0x7efdf2bc788f] 2019-10-17T20:48:49.272 INFO:tasks.ceph.osd.4.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:48:49.272 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-17T20:48:49.281 INFO:tasks.ceph.osd.4.smithi204.stderr: 0> 2019-10-17T20:48:49.264+0000 7efdc46f0700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:48:49.281 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7efdc46f0700 thread_name:tp_osd_tp 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7efdf3e32890] 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55a05769d16e] 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55a0576fa20c] 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55a0576fa89f] 2019-10-17T20:48:49.282 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55a0576ff598] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55a05770ab71] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55a0575913cb] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55a0577f6397] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55a0575acd3c] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55a057be58cc] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a057be8a40] 2019-10-17T20:48:49.283 INFO:tasks.ceph.osd.4.smithi204.stderr: 12: (()+0x76db) [0x7efdf3e276db] 2019-10-17T20:48:49.284 INFO:tasks.ceph.osd.4.smithi204.stderr: 13: (clone()+0x3f) [0x7efdf2bc788f] 2019-10-17T20:48:49.284 INFO:tasks.ceph.osd.4.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:48:49.284 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-17T20:48:49.292 INFO:tasks.ceph.osd.4.smithi204.stderr: 0> 2019-10-17T20:48:49.264+0000 7efdc46f0700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: in thread 7efdc46f0700 thread_name:tp_osd_tp 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: 1: (()+0x12890) [0x7efdf3e32890] 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55a05769d16e] 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55a0576fa20c] 2019-10-17T20:48:49.293 INFO:tasks.ceph.osd.4.smithi204.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55a0576fa89f] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55a0576ff598] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55a05770ab71] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55a0575913cb] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55a0577f6397] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55a0575acd3c] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55a057be58cc] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a057be8a40] 2019-10-17T20:48:49.294 INFO:tasks.ceph.osd.4.smithi204.stderr: 12: (()+0x76db) [0x7efdf3e276db] 2019-10-17T20:48:49.295 INFO:tasks.ceph.osd.4.smithi204.stderr: 13: (clone()+0x3f) [0x7efdf2bc788f] 2019-10-17T20:48:49.295 INFO:tasks.ceph.osd.4.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:48:49.295 INFO:tasks.ceph.osd.4.smithi204.stderr: 2019-10-17T20:48:49.491 INFO:tasks.ceph.osd.4.smithi204.stderr:daemon-helper: command crashed with signal 6 2019-10-17T20:48:49.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:49.980+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:49.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:49.980+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:51.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:51.024+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:51.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:51.024+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:52.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:52.008+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:52.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:52.008+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:52.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:52.984+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:52.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:52.984+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:53.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:53.936+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:53.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:53.936+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:54.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2019-10-17T20:48:54.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:54.964+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:54.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:54.964+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:55.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:55.972+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:55.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:55.972+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:56.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:56.940+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:56.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:56.940+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:57.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:57.916+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:57.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:57.916+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:58.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:58.944+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:58.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:58.944+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:59.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:59.968+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:48:59.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:48:59.968+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:01.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:01.008+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:01.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:01.008+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:01.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2019-10-17T20:49:02.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:02.048+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:02.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:02.048+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:02.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:02.516+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:03.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:03.064+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:03.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:03.064+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:04.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:04.072+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:04.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:04.072+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:05.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:05.116+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:05.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:05.116+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:06.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:06.096+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:06.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:06.096+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:07.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:07.116+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:07.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:07.116+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:07.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~12s 2019-10-17T20:49:07.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:07.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:08.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:08.136+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:08.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:08.136+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:09.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:09.116+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:09.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:09.116+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:10.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:10.152+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:10.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:10.152+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:11.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:11.180+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:11.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:11.180+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:12.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:12.208+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:12.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:12.208+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:12.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:12.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:13.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:13.260+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:13.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:13.260+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:13.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~19s 2019-10-17T20:49:14.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:14.292+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:14.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:14.292+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:15.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:15.284+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:15.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:15.284+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:16.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:16.264+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:16.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:16.264+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:17.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:17.244+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:17.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:17.244+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:17.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:17.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:18.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:18.204+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:18.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:18.204+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:19.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:19.212+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:19.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:19.212+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:19.679 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~25s 2019-10-17T20:49:20.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:20.248+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:20.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:20.248+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:21.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:21.240+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:21.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:21.240+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:22.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:22.284+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:22.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:22.284+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:22.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:22.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:23.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:23.328+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:23.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:23.328+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:24.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:24.328+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:24.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:24.328+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:25.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:25.376+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:25.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:25.376+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:25.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~31s 2019-10-17T20:49:26.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:26.408+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:26.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:26.408+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:27.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:27.436+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:27.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:27.436+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:27.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:27.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:28.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:28.480+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:28.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:28.480+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:29.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:29.436+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:29.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:29.436+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:30.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:30.484+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:30.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:30.484+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:31.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:31.440+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:31.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:31.440+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:32.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~37s 2019-10-17T20:49:32.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:32.488+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:32.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:32.488+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:32.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:32.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:33.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:33.496+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:33.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:33.496+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:34.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:34.524+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:34.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:34.524+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:35.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:35.564+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:35.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:35.564+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:36.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:36.608+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:36.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:36.608+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:37.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:37.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:37.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:37.632+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:37.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:37.632+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:38.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~43s 2019-10-17T20:49:38.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:38.588+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:38.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:38.588+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:39.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:39.624+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:39.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:39.624+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:40.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:40.576+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:40.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:40.576+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:41.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:41.624+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:41.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:41.624+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:42.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:42.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:42.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:42.624+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:42.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:42.624+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:43.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:43.632+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:43.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:43.632+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:44.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2019-10-17T20:49:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:44.648+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:44.648+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:45.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:45.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:46.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:46.652+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:47.524 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:47.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:47.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:47.632+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:48.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:48.608+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:49.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:49.644+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:50.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:50.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:50.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~56s 2019-10-17T20:49:51.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:51.640+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:52.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:52.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:52.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:52.596+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:53.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:53.608+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:54.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:54.644+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:55.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:55.600+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:56.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:56.584+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:56.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~62s 2019-10-17T20:49:57.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:49:57.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:49:57.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:57.576+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:58.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:58.596+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:49:59.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:49:59.560+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:00.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:00.560+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:01.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:01.568+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:02.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:02.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:02.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:02.604+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:03.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~68s 2019-10-17T20:50:03.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:03.568+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:04.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:04.604+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:05.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:05.604+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:06.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:06.604+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:07.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:07.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:07.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:07.604+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:08.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:08.640+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:09.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~75s 2019-10-17T20:50:09.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:09.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:10.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:10.708+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:10.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:10.708+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:50:10.065568+0000) 2019-10-17T20:50:11.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:11.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:11.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:11.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:50:10.065568+0000) 2019-10-17T20:50:12.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:12.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:12.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:12.672+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:12.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:12.672+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:50:10.065568+0000) 2019-10-17T20:50:13.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:13.668+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:13.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:13.668+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:50:10.065568+0000) 2019-10-17T20:50:14.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:14.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:14.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:14.660+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:50:10.065568+0000) 2019-10-17T20:50:15.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~81s 2019-10-17T20:50:15.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:15.640+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:15.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:15.640+0000 7fd46a48c700 -1 osd.1 24 heartbeat_check: no reply from 172.21.15.204:6828 osd.6 since back 2019-10-17T20:47:58.841476+0000 front 2019-10-17T20:47:58.841618+0000 (oldest deadline 2019-10-17T20:50:10.065568+0000) 2019-10-17T20:50:15.669 INFO:tasks.ceph.osd.6.smithi204.stderr:*** Caught signal (Aborted) ** 2019-10-17T20:50:15.669 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7f22d899f700 thread_name:tp_osd_tp 2019-10-17T20:50:15.672 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:15.672 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7f23070df890] 2019-10-17T20:50:15.672 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (()+0x1110d) [0x7f23070de10d] 2019-10-17T20:50:15.672 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (pthread_mutex_lock()+0x83) [0x7f23070d7023] 2019-10-17T20:50:15.672 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (PG::lock(bool) const+0x33) [0x559b790842f3] 2019-10-17T20:50:15.672 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x3bd) [0x559b790047ed] 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x559b7963d8cc] 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x559b79640a40] 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (()+0x76db) [0x7f23070d46db] 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr: 9: (clone()+0x3f) [0x7f2305e7488f] 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr:2019-10-17T20:50:15.668+0000 7f22d899f700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7f22d899f700 thread_name:tp_osd_tp 2019-10-17T20:50:15.673 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7f23070df890] 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (()+0x1110d) [0x7f23070de10d] 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (pthread_mutex_lock()+0x83) [0x7f23070d7023] 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (PG::lock(bool) const+0x33) [0x559b790842f3] 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x3bd) [0x559b790047ed] 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x559b7963d8cc] 2019-10-17T20:50:15.674 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x559b79640a40] 2019-10-17T20:50:15.675 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (()+0x76db) [0x7f23070d46db] 2019-10-17T20:50:15.675 INFO:tasks.ceph.osd.6.smithi204.stderr: 9: (clone()+0x3f) [0x7f2305e7488f] 2019-10-17T20:50:15.675 INFO:tasks.ceph.osd.6.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:50:15.675 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-17T20:50:15.680 INFO:tasks.ceph.osd.6.smithi204.stderr: -9954> 2019-10-17T20:47:30.072+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.680 INFO:tasks.ceph.osd.6.smithi204.stderr: -9889> 2019-10-17T20:47:31.064+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.680 INFO:tasks.ceph.osd.6.smithi204.stderr: -9781> 2019-10-17T20:47:32.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.680 INFO:tasks.ceph.osd.6.smithi204.stderr: -9750> 2019-10-17T20:47:33.104+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.680 INFO:tasks.ceph.osd.6.smithi204.stderr: -9658> 2019-10-17T20:47:34.064+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.690 INFO:tasks.ceph.osd.6.smithi204.stderr: -9598> 2019-10-17T20:47:35.020+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.690 INFO:tasks.ceph.osd.6.smithi204.stderr: -9557> 2019-10-17T20:47:36.040+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9528> 2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9524> 2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9522> 2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9445> 2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9441> 2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9439> 2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9413> 2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.691 INFO:tasks.ceph.osd.6.smithi204.stderr: -9409> 2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9407> 2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9323> 2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9319> 2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9317> 2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9297> 2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9293> 2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.692 INFO:tasks.ceph.osd.6.smithi204.stderr: -9291> 2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9170> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9166> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9164> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9162> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9160> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9124> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.693 INFO:tasks.ceph.osd.6.smithi204.stderr: -9120> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9118> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9116> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9114> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9092> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9088> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9086> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.694 INFO:tasks.ceph.osd.6.smithi204.stderr: -9084> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.695 INFO:tasks.ceph.osd.6.smithi204.stderr: -9082> 2019-10-17T20:47:44.012+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.698 INFO:tasks.ceph.osd.6.smithi204.stderr: 0> 2019-10-17T20:50:15.668+0000 7f22d899f700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:50:15.698 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7f22d899f700 thread_name:tp_osd_tp 2019-10-17T20:50:15.699 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-17T20:50:15.699 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:15.699 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7f23070df890] 2019-10-17T20:50:15.700 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (()+0x1110d) [0x7f23070de10d] 2019-10-17T20:50:15.700 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (pthread_mutex_lock()+0x83) [0x7f23070d7023] 2019-10-17T20:50:15.700 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (PG::lock(bool) const+0x33) [0x559b790842f3] 2019-10-17T20:50:15.700 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x3bd) [0x559b790047ed] 2019-10-17T20:50:15.700 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x559b7963d8cc] 2019-10-17T20:50:15.700 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x559b79640a40] 2019-10-17T20:50:15.701 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (()+0x76db) [0x7f23070d46db] 2019-10-17T20:50:15.701 INFO:tasks.ceph.osd.6.smithi204.stderr: 9: (clone()+0x3f) [0x7f2305e7488f] 2019-10-17T20:50:15.701 INFO:tasks.ceph.osd.6.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:50:15.701 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-17T20:50:15.701 INFO:tasks.ceph.osd.6.smithi204.stderr: -9954> 2019-10-17T20:47:30.072+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.701 INFO:tasks.ceph.osd.6.smithi204.stderr: -9889> 2019-10-17T20:47:31.064+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9781> 2019-10-17T20:47:32.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9750> 2019-10-17T20:47:33.104+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9658> 2019-10-17T20:47:34.064+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9598> 2019-10-17T20:47:35.020+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9557> 2019-10-17T20:47:36.040+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9528> 2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.702 INFO:tasks.ceph.osd.6.smithi204.stderr: -9524> 2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9522> 2019-10-17T20:47:37.084+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9445> 2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9441> 2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9439> 2019-10-17T20:47:38.036+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9413> 2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9409> 2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.703 INFO:tasks.ceph.osd.6.smithi204.stderr: -9407> 2019-10-17T20:47:39.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.704 INFO:tasks.ceph.osd.6.smithi204.stderr: -9323> 2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.704 INFO:tasks.ceph.osd.6.smithi204.stderr: -9319> 2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.704 INFO:tasks.ceph.osd.6.smithi204.stderr: -9317> 2019-10-17T20:47:40.044+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.704 INFO:tasks.ceph.osd.6.smithi204.stderr: -9297> 2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.704 INFO:tasks.ceph.osd.6.smithi204.stderr: -9293> 2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.705 INFO:tasks.ceph.osd.6.smithi204.stderr: -9291> 2019-10-17T20:47:41.000+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.705 INFO:tasks.ceph.osd.6.smithi204.stderr: -9170> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.705 INFO:tasks.ceph.osd.6.smithi204.stderr: -9166> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.706 INFO:tasks.ceph.osd.6.smithi204.stderr: -9164> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.706 INFO:tasks.ceph.osd.6.smithi204.stderr: -9162> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.706 INFO:tasks.ceph.osd.6.smithi204.stderr: -9160> 2019-10-17T20:47:42.028+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.706 INFO:tasks.ceph.osd.6.smithi204.stderr: -9124> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.707 INFO:tasks.ceph.osd.6.smithi204.stderr: -9120> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.707 INFO:tasks.ceph.osd.6.smithi204.stderr: -9118> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.707 INFO:tasks.ceph.osd.6.smithi204.stderr: -9116> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.707 INFO:tasks.ceph.osd.6.smithi204.stderr: -9114> 2019-10-17T20:47:43.060+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.707 INFO:tasks.ceph.osd.6.smithi204.stderr: -9092> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.854313+0000 front 2019-10-17T20:47:15.854412+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.707 INFO:tasks.ceph.osd.6.smithi204.stderr: -9088> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.854445+0000 front 2019-10-17T20:47:15.854475+0000 (oldest deadline 2019-10-17T20:47:36.354411+0000) 2019-10-17T20:50:15.708 INFO:tasks.ceph.osd.6.smithi204.stderr: -9086> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:29.750455+0000 front 2019-10-17T20:46:29.750657+0000 (oldest deadline 2019-10-17T20:46:55.650445+0000) 2019-10-17T20:50:15.708 INFO:tasks.ceph.osd.6.smithi204.stderr: -9084> 2019-10-17T20:47:44.008+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:16.354710+0000 front 2019-10-17T20:47:16.354872+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.708 INFO:tasks.ceph.osd.6.smithi204.stderr: -9082> 2019-10-17T20:47:44.012+0000 7f22fb3c2700 -1 osd.6 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:16.354772+0000 front 2019-10-17T20:47:16.355048+0000 (oldest deadline 2019-10-17T20:47:41.054679+0000) 2019-10-17T20:50:15.709 INFO:tasks.ceph.osd.6.smithi204.stderr: 0> 2019-10-17T20:50:15.668+0000 7f22d899f700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:50:15.709 INFO:tasks.ceph.osd.6.smithi204.stderr: in thread 7f22d899f700 thread_name:tp_osd_tp 2019-10-17T20:50:15.709 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: 1: (()+0x12890) [0x7f23070df890] 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: 2: (()+0x1110d) [0x7f23070de10d] 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: 3: (pthread_mutex_lock()+0x83) [0x7f23070d7023] 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: 4: (PG::lock(bool) const+0x33) [0x559b790842f3] 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: 5: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x3bd) [0x559b790047ed] 2019-10-17T20:50:15.710 INFO:tasks.ceph.osd.6.smithi204.stderr: 6: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x559b7963d8cc] 2019-10-17T20:50:15.711 INFO:tasks.ceph.osd.6.smithi204.stderr: 7: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x559b79640a40] 2019-10-17T20:50:15.711 INFO:tasks.ceph.osd.6.smithi204.stderr: 8: (()+0x76db) [0x7f23070d46db] 2019-10-17T20:50:15.711 INFO:tasks.ceph.osd.6.smithi204.stderr: 9: (clone()+0x3f) [0x7f2305e7488f] 2019-10-17T20:50:15.711 INFO:tasks.ceph.osd.6.smithi204.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:50:15.711 INFO:tasks.ceph.osd.6.smithi204.stderr: 2019-10-17T20:50:15.801 INFO:tasks.ceph.osd.6.smithi204.stderr:daemon-helper: command crashed with signal 6 2019-10-17T20:50:16.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:16.688+0000 7fd46a48c700 -1 osd.1 25 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:48:23.541376+0000) 2019-10-17T20:50:16.769 INFO:tasks.ceph.osd.2.smithi092.stderr:*** Caught signal (Aborted) ** 2019-10-17T20:50:16.769 INFO:tasks.ceph.osd.2.smithi092.stderr: in thread 7f6510321700 thread_name:tp_osd_tp 2019-10-17T20:50:16.775 INFO:tasks.ceph.osd.2.smithi092.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:16.775 INFO:tasks.ceph.osd.2.smithi092.stderr: 1: (()+0x12890) [0x7f653e260890] 2019-10-17T20:50:16.775 INFO:tasks.ceph.osd.2.smithi092.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55c89c55616e] 2019-10-17T20:50:16.775 INFO:tasks.ceph.osd.2.smithi092.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55c89c5b320c] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55c89c5b389f] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55c89c5b8598] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55c89c5c3b71] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55c89c44a3cb] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55c89c6af397] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55c89c465d3c] 2019-10-17T20:50:16.776 INFO:tasks.ceph.osd.2.smithi092.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55c89ca9e8cc] 2019-10-17T20:50:16.777 INFO:tasks.ceph.osd.2.smithi092.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c89caa1a40] 2019-10-17T20:50:16.777 INFO:tasks.ceph.osd.2.smithi092.stderr: 12: (()+0x76db) [0x7f653e2556db] 2019-10-17T20:50:16.777 INFO:tasks.ceph.osd.2.smithi092.stderr: 13: (clone()+0x3f) [0x7f653cff588f] 2019-10-17T20:50:16.777 INFO:tasks.ceph.osd.2.smithi092.stderr:2019-10-17T20:50:16.776+0000 7f6510321700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:50:16.777 INFO:tasks.ceph.osd.2.smithi092.stderr: in thread 7f6510321700 thread_name:tp_osd_tp 2019-10-17T20:50:16.777 INFO:tasks.ceph.osd.2.smithi092.stderr: 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: 1: (()+0x12890) [0x7f653e260890] 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55c89c55616e] 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55c89c5b320c] 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55c89c5b389f] 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55c89c5b8598] 2019-10-17T20:50:16.778 INFO:tasks.ceph.osd.2.smithi092.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55c89c5c3b71] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55c89c44a3cb] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55c89c6af397] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55c89c465d3c] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55c89ca9e8cc] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c89caa1a40] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 12: (()+0x76db) [0x7f653e2556db] 2019-10-17T20:50:16.779 INFO:tasks.ceph.osd.2.smithi092.stderr: 13: (clone()+0x3f) [0x7f653cff588f] 2019-10-17T20:50:16.780 INFO:tasks.ceph.osd.2.smithi092.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:50:16.780 INFO:tasks.ceph.osd.2.smithi092.stderr: 2019-10-17T20:50:16.793 INFO:tasks.ceph.osd.2.smithi092.stderr: -9970> 2019-10-17T20:46:57.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.793 INFO:tasks.ceph.osd.2.smithi092.stderr: -9928> 2019-10-17T20:46:58.988+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.793 INFO:tasks.ceph.osd.2.smithi092.stderr: -9823> 2019-10-17T20:46:59.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.793 INFO:tasks.ceph.osd.2.smithi092.stderr: -9787> 2019-10-17T20:47:00.964+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.793 INFO:tasks.ceph.osd.2.smithi092.stderr: -9749> 2019-10-17T20:47:01.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.793 INFO:tasks.ceph.osd.2.smithi092.stderr: -9718> 2019-10-17T20:47:03.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.794 INFO:tasks.ceph.osd.2.smithi092.stderr: -9677> 2019-10-17T20:47:04.068+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.794 INFO:tasks.ceph.osd.2.smithi092.stderr: -9649> 2019-10-17T20:47:05.108+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.794 INFO:tasks.ceph.osd.2.smithi092.stderr: -9572> 2019-10-17T20:47:06.072+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.794 INFO:tasks.ceph.osd.2.smithi092.stderr: -9524> 2019-10-17T20:47:07.080+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.794 INFO:tasks.ceph.osd.2.smithi092.stderr: -9458> 2019-10-17T20:47:08.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.794 INFO:tasks.ceph.osd.2.smithi092.stderr: -9434> 2019-10-17T20:47:08.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -9334> 2019-10-17T20:47:10.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -9308> 2019-10-17T20:47:10.984+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -9219> 2019-10-17T20:47:12.024+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -9177> 2019-10-17T20:47:13.044+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -9147> 2019-10-17T20:47:14.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -9111> 2019-10-17T20:47:15.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.795 INFO:tasks.ceph.osd.2.smithi092.stderr: -8993> 2019-10-17T20:47:16.064+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8959> 2019-10-17T20:47:17.068+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8912> 2019-10-17T20:47:18.072+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8888> 2019-10-17T20:47:19.060+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8833> 2019-10-17T20:47:20.008+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8767> 2019-10-17T20:47:21.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8711> 2019-10-17T20:47:22.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.796 INFO:tasks.ceph.osd.2.smithi092.stderr: -8680> 2019-10-17T20:47:23.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8620> 2019-10-17T20:47:24.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8590> 2019-10-17T20:47:24.968+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8525> 2019-10-17T20:47:25.960+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8492> 2019-10-17T20:47:26.984+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8457> 2019-10-17T20:47:28.028+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8433> 2019-10-17T20:47:29.020+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.797 INFO:tasks.ceph.osd.2.smithi092.stderr: -8397> 2019-10-17T20:47:29.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8337> 2019-10-17T20:47:30.964+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8268> 2019-10-17T20:47:31.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8233> 2019-10-17T20:47:33.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8167> 2019-10-17T20:47:33.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8131> 2019-10-17T20:47:35.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8101> 2019-10-17T20:47:35.976+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.798 INFO:tasks.ceph.osd.2.smithi092.stderr: -8033> 2019-10-17T20:47:37.028+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7980> 2019-10-17T20:47:38.000+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7950> 2019-10-17T20:47:38.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7904> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7901> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7899> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7897> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.799 INFO:tasks.ceph.osd.2.smithi092.stderr: -7894> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7871> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7868> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7866> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7864> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7861> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7810> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.800 INFO:tasks.ceph.osd.2.smithi092.stderr: -7807> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7805> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7803> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7800> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7768> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7765> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7763> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.801 INFO:tasks.ceph.osd.2.smithi092.stderr: -7761> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.802 INFO:tasks.ceph.osd.2.smithi092.stderr: -7758> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.802 INFO:tasks.ceph.osd.2.smithi092.stderr: -7727> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.802 INFO:tasks.ceph.osd.2.smithi092.stderr: -7724> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.802 INFO:tasks.ceph.osd.2.smithi092.stderr: -7722> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.802 INFO:tasks.ceph.osd.2.smithi092.stderr: -7720> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.802 INFO:tasks.ceph.osd.2.smithi092.stderr: -7717> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.805 INFO:tasks.ceph.osd.2.smithi092.stderr: 0> 2019-10-17T20:50:16.776+0000 7f6510321700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:50:16.805 INFO:tasks.ceph.osd.2.smithi092.stderr: in thread 7f6510321700 thread_name:tp_osd_tp 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: 1: (()+0x12890) [0x7f653e260890] 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55c89c55616e] 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55c89c5b320c] 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55c89c5b389f] 2019-10-17T20:50:16.806 INFO:tasks.ceph.osd.2.smithi092.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55c89c5b8598] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55c89c5c3b71] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55c89c44a3cb] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55c89c6af397] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55c89c465d3c] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55c89ca9e8cc] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c89caa1a40] 2019-10-17T20:50:16.807 INFO:tasks.ceph.osd.2.smithi092.stderr: 12: (()+0x76db) [0x7f653e2556db] 2019-10-17T20:50:16.808 INFO:tasks.ceph.osd.2.smithi092.stderr: 13: (clone()+0x3f) [0x7f653cff588f] 2019-10-17T20:50:16.808 INFO:tasks.ceph.osd.2.smithi092.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:50:16.808 INFO:tasks.ceph.osd.2.smithi092.stderr: 2019-10-17T20:50:16.808 INFO:tasks.ceph.osd.2.smithi092.stderr: -9970> 2019-10-17T20:46:57.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.808 INFO:tasks.ceph.osd.2.smithi092.stderr: -9928> 2019-10-17T20:46:58.988+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.808 INFO:tasks.ceph.osd.2.smithi092.stderr: -9823> 2019-10-17T20:46:59.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9787> 2019-10-17T20:47:00.964+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9749> 2019-10-17T20:47:01.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9718> 2019-10-17T20:47:03.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9677> 2019-10-17T20:47:04.068+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9649> 2019-10-17T20:47:05.108+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9572> 2019-10-17T20:47:06.072+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.809 INFO:tasks.ceph.osd.2.smithi092.stderr: -9524> 2019-10-17T20:47:07.080+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.810 INFO:tasks.ceph.osd.2.smithi092.stderr: -9458> 2019-10-17T20:47:08.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.810 INFO:tasks.ceph.osd.2.smithi092.stderr: -9434> 2019-10-17T20:47:08.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.810 INFO:tasks.ceph.osd.2.smithi092.stderr: -9334> 2019-10-17T20:47:10.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.811 INFO:tasks.ceph.osd.2.smithi092.stderr: -9308> 2019-10-17T20:47:10.984+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.811 INFO:tasks.ceph.osd.2.smithi092.stderr: -9219> 2019-10-17T20:47:12.024+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.811 INFO:tasks.ceph.osd.2.smithi092.stderr: -9177> 2019-10-17T20:47:13.044+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.812 INFO:tasks.ceph.osd.2.smithi092.stderr: -9147> 2019-10-17T20:47:14.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.812 INFO:tasks.ceph.osd.2.smithi092.stderr: -9111> 2019-10-17T20:47:15.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.814 INFO:tasks.ceph.osd.2.smithi092.stderr: -8993> 2019-10-17T20:47:16.064+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.814 INFO:tasks.ceph.osd.2.smithi092.stderr: -8959> 2019-10-17T20:47:17.068+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.814 INFO:tasks.ceph.osd.2.smithi092.stderr: -8912> 2019-10-17T20:47:18.072+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.815 INFO:tasks.ceph.osd.2.smithi092.stderr: -8888> 2019-10-17T20:47:19.060+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.816 INFO:tasks.ceph.osd.2.smithi092.stderr: -8833> 2019-10-17T20:47:20.008+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.816 INFO:tasks.ceph.osd.2.smithi092.stderr: -8767> 2019-10-17T20:47:21.004+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.828 INFO:tasks.ceph.osd.2.smithi092.stderr: -8711> 2019-10-17T20:47:22.012+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.828 INFO:tasks.ceph.osd.2.smithi092.stderr: -8680> 2019-10-17T20:47:23.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.828 INFO:tasks.ceph.osd.2.smithi092.stderr: -8620> 2019-10-17T20:47:24.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8590> 2019-10-17T20:47:24.968+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8525> 2019-10-17T20:47:25.960+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8492> 2019-10-17T20:47:26.984+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8457> 2019-10-17T20:47:28.028+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8433> 2019-10-17T20:47:29.020+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8397> 2019-10-17T20:47:29.996+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.829 INFO:tasks.ceph.osd.2.smithi092.stderr: -8337> 2019-10-17T20:47:30.964+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -8268> 2019-10-17T20:47:31.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -8233> 2019-10-17T20:47:33.036+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -8167> 2019-10-17T20:47:33.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -8131> 2019-10-17T20:47:35.016+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -8101> 2019-10-17T20:47:35.976+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -8033> 2019-10-17T20:47:37.028+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.830 INFO:tasks.ceph.osd.2.smithi092.stderr: -7980> 2019-10-17T20:47:38.000+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7950> 2019-10-17T20:47:38.992+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7904> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7901> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7899> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7897> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7894> 2019-10-17T20:47:39.972+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.831 INFO:tasks.ceph.osd.2.smithi092.stderr: -7871> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7868> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7866> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7864> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7861> 2019-10-17T20:47:40.948+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7810> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7807> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.832 INFO:tasks.ceph.osd.2.smithi092.stderr: -7805> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7803> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7800> 2019-10-17T20:47:41.908+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7768> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7765> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7763> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7761> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.833 INFO:tasks.ceph.osd.2.smithi092.stderr: -7758> 2019-10-17T20:47:42.872+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.834 INFO:tasks.ceph.osd.2.smithi092.stderr: -7727> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6823 osd.0 since back 2019-10-17T20:47:15.046104+0000 front 2019-10-17T20:47:15.045714+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.834 INFO:tasks.ceph.osd.2.smithi092.stderr: -7724> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.92:6806 osd.3 since back 2019-10-17T20:47:15.045630+0000 front 2019-10-17T20:47:15.046176+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.834 INFO:tasks.ceph.osd.2.smithi092.stderr: -7722> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6804 osd.4 since back 2019-10-17T20:46:28.342966+0000 front 2019-10-17T20:46:28.342879+0000 (oldest deadline 2019-10-17T20:46:53.042179+0000) 2019-10-17T20:50:16.834 INFO:tasks.ceph.osd.2.smithi092.stderr: -7720> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6820 osd.5 since back 2019-10-17T20:47:15.045608+0000 front 2019-10-17T20:47:15.046130+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.834 INFO:tasks.ceph.osd.2.smithi092.stderr: -7717> 2019-10-17T20:47:43.820+0000 7f6532543700 -1 osd.2 21 heartbeat_check: no reply from 172.21.15.204:6812 osd.7 since back 2019-10-17T20:47:15.045691+0000 front 2019-10-17T20:47:15.045588+0000 (oldest deadline 2019-10-17T20:47:39.145082+0000) 2019-10-17T20:50:16.836 INFO:tasks.ceph.osd.2.smithi092.stderr: 0> 2019-10-17T20:50:16.776+0000 7f6510321700 -1 *** Caught signal (Aborted) ** 2019-10-17T20:50:16.836 INFO:tasks.ceph.osd.2.smithi092.stderr: in thread 7f6510321700 thread_name:tp_osd_tp 2019-10-17T20:50:16.836 INFO:tasks.ceph.osd.2.smithi092.stderr: 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: ceph version 15.0.0-6144-gf0009d4 (f0009d4765d23f6e6a0d55d83c1e9f63c45550d5) octopus (dev) 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: 1: (()+0x12890) [0x7f653e260890] 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: 2: (PrimaryLogPG::finish_ctx(PrimaryLogPG::OpContext*, int, int)+0x30e) [0x55c89c55616e] 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: 3: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x2ac) [0x55c89c5b320c] 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: 4: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3bf) [0x55c89c5b389f] 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: 5: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x38e8) [0x55c89c5b8598] 2019-10-17T20:50:16.837 INFO:tasks.ceph.osd.2.smithi092.stderr: 6: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xcf1) [0x55c89c5c3b71] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x17b) [0x55c89c44a3cb] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 8: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x67) [0x55c89c6af397] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55c89c465d3c] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55c89ca9e8cc] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c89caa1a40] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 12: (()+0x76db) [0x7f653e2556db] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: 13: (clone()+0x3f) [0x7f653cff588f] 2019-10-17T20:50:16.838 INFO:tasks.ceph.osd.2.smithi092.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-10-17T20:50:16.839 INFO:tasks.ceph.osd.2.smithi092.stderr: 2019-10-17T20:50:16.976 INFO:tasks.ceph.osd.2.smithi092.stderr:daemon-helper: command crashed with signal 6 2019-10-17T20:50:17.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:17.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:21.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2019-10-17T20:50:21.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~87s 2019-10-17T20:50:21.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2019-10-17T20:50:22.529 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:22.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:27.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:27.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:27.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2019-10-17T20:50:27.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~93s 2019-10-17T20:50:27.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2019-10-17T20:50:32.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:32.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:33.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~12s 2019-10-17T20:50:33.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~99s 2019-10-17T20:50:33.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~12s 2019-10-17T20:50:37.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:37.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:39.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~18s 2019-10-17T20:50:39.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~105s 2019-10-17T20:50:39.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~18s 2019-10-17T20:50:40.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:40.492+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:41.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:41.516+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:42.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:42.516+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:42.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:42.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:43.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:43.524+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:44.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:44.564+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:45.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:45.568+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:45.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~24s 2019-10-17T20:50:45.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~111s 2019-10-17T20:50:45.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~24s 2019-10-17T20:50:46.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:46.540+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:47.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:47.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:47.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:47.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:48.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:48.508+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:49.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:49.508+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:50.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:50.540+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:51.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:51.572+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:51.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~30s 2019-10-17T20:50:51.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~117s 2019-10-17T20:50:51.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~30s 2019-10-17T20:50:52.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:52.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:52.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:52.580+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:53.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:53.556+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:54.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:54.572+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:55.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:55.600+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:56.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:56.616+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:57.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:50:57.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:50:57.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:57.572+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:57.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~36s 2019-10-17T20:50:57.659 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~123s 2019-10-17T20:50:57.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~36s 2019-10-17T20:50:58.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:58.532+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:50:59.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:50:59.500+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:00.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:00.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:01.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:01.520+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:02.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:02.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:02.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:02.564+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:03.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:03.532+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:03.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~42s 2019-10-17T20:51:03.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~129s 2019-10-17T20:51:03.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~42s 2019-10-17T20:51:04.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:04.488+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:05.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:05.476+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:06.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:06.504+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:07.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:07.516+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:07.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:07.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:08.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:08.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:09.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:09.544+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:09.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~48s 2019-10-17T20:51:09.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~135s 2019-10-17T20:51:09.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~48s 2019-10-17T20:51:10.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:10.572+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:11.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:11.584+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:12.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:12.540+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:12.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:12.556+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:13.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:13.548+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:14.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:14.560+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:15.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:15.564+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:15.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~54s 2019-10-17T20:51:15.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~141s 2019-10-17T20:51:15.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~54s 2019-10-17T20:51:16.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:16.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:17.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:17.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:17.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:17.540+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:18.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:18.504+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:19.544+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:20.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:20.532+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:21.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:21.564+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:21.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~60s 2019-10-17T20:51:21.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~147s 2019-10-17T20:51:21.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~60s 2019-10-17T20:51:22.537 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:22.540+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:22.544+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:23.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:23.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:24.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:24.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:25.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:25.556+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:26.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:26.592+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:27.537 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:27.540+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:27.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:27.616+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:27.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~66s 2019-10-17T20:51:27.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~153s 2019-10-17T20:51:27.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~66s 2019-10-17T20:51:28.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:28.568+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:29.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:29.552+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:30.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:30.576+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:31.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:31.584+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:32.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:32.540+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:32.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:32.560+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:33.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:33.576+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:33.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~72s 2019-10-17T20:51:33.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~159s 2019-10-17T20:51:33.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~72s 2019-10-17T20:51:34.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:34.592+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:35.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:35.612+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:36.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:36.600+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:37.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:37.596+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:37.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:37.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:38.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:38.600+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:39.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:39.648+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:39.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~78s 2019-10-17T20:51:39.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~165s 2019-10-17T20:51:39.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~78s 2019-10-17T20:51:40.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:40.624+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:41.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:41.604+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:42.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:42.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:42.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:42.572+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:43.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:43.540+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:44.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:44.560+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:45.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:45.600+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:45.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~84s 2019-10-17T20:51:45.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~171s 2019-10-17T20:51:45.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~84s 2019-10-17T20:51:46.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:46.556+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:47.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:47.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:47.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:47.564+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:48.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:48.548+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:49.544+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:50.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:50.500+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:51.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:51.552+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:51.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2019-10-17T20:51:51.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~177s 2019-10-17T20:51:51.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~90s 2019-10-17T20:51:52.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:52.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:52.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:52.584+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:53.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:53.536+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:54.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:54.540+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:55.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:55.588+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:56.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:56.588+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:57.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:51:57.543+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:51:57.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:57.603+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:57.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~96s 2019-10-17T20:51:57.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~183s 2019-10-17T20:51:57.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~96s 2019-10-17T20:51:58.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:58.571+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:51:59.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:51:59.567+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:00.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:00.534+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:01.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:01.578+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:02.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:02.534+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:02.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:02.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:03.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:03.486+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:03.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~102s 2019-10-17T20:52:03.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~189s 2019-10-17T20:52:03.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~102s 2019-10-17T20:52:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:04.521+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:05.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:05.569+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:06.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:06.565+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:07.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:07.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:07.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:07.585+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:08.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:08.612+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:09.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:09.608+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:09.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~108s 2019-10-17T20:52:09.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~195s 2019-10-17T20:52:09.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~108s 2019-10-17T20:52:10.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:10.608+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:11.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:11.636+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:12.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:12.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:12.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:12.600+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:13.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:13.571+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:14.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:14.607+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:15.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:15.635+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:15.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~114s 2019-10-17T20:52:15.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~201s 2019-10-17T20:52:15.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~114s 2019-10-17T20:52:16.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:16.591+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:17.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:17.547+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:17.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:17.627+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:18.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:18.650+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:19.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:19.650+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:20.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:20.650+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:21.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:21.674+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:21.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~120s 2019-10-17T20:52:21.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~207s 2019-10-17T20:52:21.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~120s 2019-10-17T20:52:22.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:22.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:22.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:22.666+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:23.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:23.650+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:24.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:24.641+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:25.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:25.657+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:26.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:26.705+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:27.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:27.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:27.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:27.657+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:27.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~126s 2019-10-17T20:52:27.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~213s 2019-10-17T20:52:27.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~126s 2019-10-17T20:52:28.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:28.673+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:29.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:29.629+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:30.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:30.653+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:31.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:31.664+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:32.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:32.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:32.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:32.644+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:33.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:33.676+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:33.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~132s 2019-10-17T20:52:33.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~219s 2019-10-17T20:52:33.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~132s 2019-10-17T20:52:34.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:34.652+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:35.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:35.684+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:36.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:36.652+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:37.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:37.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:37.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:37.644+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:38.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:38.692+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:39.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:39.696+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:39.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2019-10-17T20:52:39.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~225s 2019-10-17T20:52:39.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~138s 2019-10-17T20:52:40.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:40.695+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:41.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:41.695+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:42.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:42.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:42.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:42.679+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:43.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:43.703+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:44.655+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:45.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:45.639+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:45.859 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~144s 2019-10-17T20:52:45.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~231s 2019-10-17T20:52:45.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~144s 2019-10-17T20:52:46.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:46.603+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:47.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:47.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:47.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:47.579+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:48.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:48.583+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:49.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:49.551+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:50.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:50.507+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:51.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:51.466+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:51.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~150s 2019-10-17T20:52:51.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~237s 2019-10-17T20:52:51.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~150s 2019-10-17T20:52:52.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:52.430+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:52.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:52.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:53.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:53.410+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:54.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:54.374+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:55.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:55.402+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:56.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:56.382+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:57.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:57.346+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:57.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:52:57.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:52:57.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~156s 2019-10-17T20:52:57.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~243s 2019-10-17T20:52:57.884 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~156s 2019-10-17T20:52:58.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:58.378+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:52:59.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:52:59.334+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:00.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:00.314+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:01.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:01.326+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:02.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:02.302+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:02.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:02.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:03.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:03.274+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:03.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~162s 2019-10-17T20:53:03.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~249s 2019-10-17T20:53:03.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~162s 2019-10-17T20:53:04.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:04.262+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:05.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:05.229+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:06.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:06.181+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:07.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:07.209+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:07.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:07.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:08.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:08.237+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:09.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:09.209+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:09.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~168s 2019-10-17T20:53:09.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~255s 2019-10-17T20:53:09.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~168s 2019-10-17T20:53:10.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:10.229+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:11.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:11.189+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:12.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:12.201+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:12.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:12.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:13.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:13.173+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:14.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:14.197+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:15.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:15.245+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:15.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~174s 2019-10-17T20:53:15.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~261s 2019-10-17T20:53:15.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~174s 2019-10-17T20:53:16.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:16.233+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:17.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:17.277+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:17.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:17.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:18.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:18.309+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:19.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:19.301+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:20.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:20.305+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:21.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:21.297+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:21.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2019-10-17T20:53:21.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~267s 2019-10-17T20:53:21.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2019-10-17T20:53:22.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:22.305+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:22.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:22.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:23.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:23.320+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:24.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:24.320+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:25.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:25.316+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:26.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:26.332+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:27.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:27.292+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:27.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:27.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:27.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~186s 2019-10-17T20:53:27.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~273s 2019-10-17T20:53:27.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~186s 2019-10-17T20:53:28.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:28.308+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:29.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:29.288+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:30.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:30.316+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:31.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:31.356+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:32.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:32.392+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:32.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:32.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:33.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:33.428+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:33.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~192s 2019-10-17T20:53:33.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~279s 2019-10-17T20:53:33.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~192s 2019-10-17T20:53:34.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:34.380+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:35.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:35.364+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:36.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:36.364+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:37.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:37.344+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:37.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:37.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:38.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:38.324+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:39.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:39.344+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:39.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~198s 2019-10-17T20:53:39.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~285s 2019-10-17T20:53:39.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~198s 2019-10-17T20:53:40.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:40.384+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:41.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:41.380+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:42.376+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:42.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:42.564+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:43.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:43.376+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:44.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:44.332+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:45.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:45.300+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:45.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~204s 2019-10-17T20:53:45.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~291s 2019-10-17T20:53:45.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~204s 2019-10-17T20:53:46.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:46.276+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:47.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:47.267+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:47.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:47.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:48.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:48.227+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:49.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:49.279+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:50.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:50.255+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:51.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:51.283+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:51.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~210s 2019-10-17T20:53:51.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~297s 2019-10-17T20:53:51.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~210s 2019-10-17T20:53:52.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:52.243+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:52.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:52.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:53.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:53.247+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:54.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:54.199+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:55.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:55.247+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:56.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:56.247+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:57.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:57.231+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:57.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:53:57.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:53:57.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~216s 2019-10-17T20:53:57.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~303s 2019-10-17T20:53:57.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~216s 2019-10-17T20:53:57.992 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-10-17T20:53:57.992 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa/tasks/daemonwatchdog.py", line 38, in _run self.watch() File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa/tasks/daemonwatchdog.py", line 114, in watch self.bark() File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sjust-testing2/qa/tasks/daemonwatchdog.py", line 54, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 253, in __getattr__ raise AttributeError(name) AttributeError: mounts 2019-10-17T20:53:58.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:58.255+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:53:59.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:53:59.211+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:00.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:00.227+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:01.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:01.267+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:02.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:02.259+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:02.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:02.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:03.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:03.215+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:04.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:04.211+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:05.259+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:06.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:06.259+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:07.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:07.303+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:07.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:07.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:08.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:08.319+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:09.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:09.367+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:10.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:10.339+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:11.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:11.375+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:12.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:12.387+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:12.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:12.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:13.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:13.399+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:14.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:14.411+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:15.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:15.363+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:16.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:16.358+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:17.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:17.358+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:17.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:17.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:18.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:18.334+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:19.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:19.382+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:20.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:20.394+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:21.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:21.442+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:22.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:22.478+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:22.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:22.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:23.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:23.498+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:24.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:24.470+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:25.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:25.510+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:26.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:26.502+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:27.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:27.542+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:27.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:27.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:28.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:28.502+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:29.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:29.482+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:30.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:30.506+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:31.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:31.502+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:32.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:32.506+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:32.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:32.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:33.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:33.482+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:34.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:34.482+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:35.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:35.486+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:36.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:36.442+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:37.402+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:37.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:37.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:38.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:38.446+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:39.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:39.478+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:40.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:40.506+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:41.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:41.522+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:42.562+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:42.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:42.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:43.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:43.518+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:44.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:44.498+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:45.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:45.534+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:46.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:46.582+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:47.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:47.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:47.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:47.614+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:48.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:48.578+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:49.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:49.574+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:50.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:50.578+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:51.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:51.621+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:52.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:52.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:52.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:52.613+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:53.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:53.621+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:54.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:54.585+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:55.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:55.617+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:56.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:56.621+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:57.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:54:57.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:54:57.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:57.589+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:58.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:58.633+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:54:59.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:54:59.657+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:00.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:00.613+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:01.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:01.621+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:02.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:02.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:02.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:02.585+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:03.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:03.581+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:04.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:04.593+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:05.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:05.597+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:06.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:06.565+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:07.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:07.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:07.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:07.613+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:08.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:08.589+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:09.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:09.589+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:10.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:10.633+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:11.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:11.589+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:12.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:12.577+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:12.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:12.577+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:13.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:13.621+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:14.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:14.577+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:15.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:15.621+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:16.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:16.573+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:17.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:17.525+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:17.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:17.577+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:18.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:18.533+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:19.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:19.577+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:19.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:19.889+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:19.893 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:19.889+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:20.093+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946480 0x5623759e5080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.096 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:20.093+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.500 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:20.497+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f63080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:20.497+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.606 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:20.601+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:20.601+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:20.609+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:20.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:20.805+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:20.809 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:20.805+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:21.212 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:21.209+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a14c00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:21.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:21.209+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a35b00 0x5623759e6680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:21.303 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:21.301+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d926f58880 0x55d926f64680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:21.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:21.301+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:21.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:21.641+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:22.016 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:22.013+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c000 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:22.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:22.013+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946480 0x5623759e5080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:22.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:22.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:22.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:22.673+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:22.908 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:22.905+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f63080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:22.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:22.905+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:23.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:23.617+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:23.620 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:23.617+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:23.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:23.681+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:24.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:24.729+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:25.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:25.684+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:26.112 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:26.108+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a35b00 0x5623759e6680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:26.112 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:26.108+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:26.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:26.664+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:26.825 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:26.821+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e880 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:26.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:26.820+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:27.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:27.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:27.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:27.616+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:28.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:28.580+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:29.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:29.584+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:29.892 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:29.889+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a14c00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:30.095 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:30.089+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:30.499 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:30.493+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6d680 0x556576a13600 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:30.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:30.596+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:30.606 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:30.600+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:30.808 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:30.804+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:31.212 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:31.208+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d926f58d00 0x55d926ec2000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:31.304 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:31.297+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:31.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:31.648+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:32.016 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:32.012+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f63080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:32.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:32.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:32.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:32.684+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:32.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:32.888+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:33.095 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:33.092+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:33.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:33.496+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a37600 0x5623759e6680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:33.606 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:33.601+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:33.606 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:33.600+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:33.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:33.656+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:33.808 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:33.804+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:33.808 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:33.805+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e880 0x556576a14c00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:34.212 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:34.208+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:34.212 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:34.209+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b12d00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:34.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:34.300+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a35b00 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:34.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:34.656+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:35.016 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:35.012+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d926f57200 0x55d926ec2580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:35.016 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:35.013+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b12d00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:35.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:35.680+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:35.892 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:35.888+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f63080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:35.893 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:35.889+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:36.096 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:36.092+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:36.099 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:36.089+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:36.500 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:36.496+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:36.501 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:36.493+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a14c00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:36.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:36.600+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:36.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:36.660+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:36.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:36.804+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e6100 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:37.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:37.208+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:37.303 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:37.300+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:37.304 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:37.297+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:37.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:37.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:37.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:37.684+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:38.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:38.012+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:38.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:38.676+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:38.893 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:38.888+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d926f56900 0x55d926f64680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:39.095 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:39.092+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f63080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:39.499 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:39.496+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:39.606 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:39.600+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:39.607 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:39.600+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:39.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:39.704+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:39.808 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:39.804+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:39.808 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:39.804+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:40.212 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:40.208+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722a880 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:40.212 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:40.208+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10480 0x556576a14c00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:40.303 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:40.300+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f63080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:40.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:40.700+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:41.016 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:41.012+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:41.016 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:41.012+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:41.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:41.716+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:41.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:41.888+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623759e6100 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:41.893 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:41.888+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b12d00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:42.095 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:42.092+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:42.096 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:42.088+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:42.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:42.496+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:42.500 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:42.496+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:42.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:42.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:42.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:42.600+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a35b00 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:42.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:42.696+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:42.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:42.804+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:43.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:43.208+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36d00 0x5623759e7700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:43.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:43.300+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:43.304 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:43.296+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:43.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:43.744+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:44.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:44.012+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:44.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:44.724+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:44.893 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:44.888+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:44.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:44.888+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a35b00 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.096 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:45.088+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.096 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:45.092+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:45.092+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.499 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:45.492+0000 7f8a0a312700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.500 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:45.496+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229f80 0x55d926ec4680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:45.496+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36d00 0x5623759e7700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:45.600+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:45.716+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:45.808 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:45.804+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:45.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:45.804+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:46.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:46.208+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a35b00 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:46.213 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:46.208+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:46.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:46.300+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:46.304 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:46.300+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:46.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:46.680+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:47.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:47.012+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376947200 0x5623756c6000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:47.017 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:47.012+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830c000 0x55d926d28680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:47.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:47.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:47.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:47.648+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:48.024 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:48.020+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:48.227 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:48.224+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:48.607 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:48.600+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6e400 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:48.631 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:48.628+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:48.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:48.688+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:48.808 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:48.804+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:49.212 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:49.208+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:49.435 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:49.432+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926d28680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:49.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:49.732+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:50.016 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:50.012+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:50.608 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:50.604+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:50.608 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:50.604+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830c480 0x55d926ec2000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:50.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:50.740+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:50.812 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:50.808+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:50.813 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:50.808+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:51.039 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:51.036+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:51.216 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:51.212+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926f62580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:51.217 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:51.212+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:51.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:51.744+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:52.020 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:52.016+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926d28680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:52.021 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:52.016+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:52.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:52.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:52.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:52.776+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:53.624 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:53.620+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:53.625 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:53.620+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830c900 0x55d926ec2580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:53.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:53.816+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:54.243 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:54.240+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:54.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:54.780+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:55.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:55.748+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:56.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:56.712+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:56.828 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:55:56.824+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:56.829 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:56.824+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:57.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:57.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:55:57.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:57.708+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:58.024 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:58.016+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b11200 0x556576347700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:58.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:58.020+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:58.227 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:58.220+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:58.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:58.224+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376947200 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:58.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:58.628+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376947200 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:58.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:58.624+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:58.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:58.696+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:55:59.436 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:55:59.428+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:59.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:55:59.432+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:55:59.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:55:59.684+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:00.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:00.603+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:00.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:00.715+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:00.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:00.807+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:01.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:01.019+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:01.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:01.211+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:01.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:01.223+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376947200 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:01.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:01.627+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376946000 0x5623759e5b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:01.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:01.711+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:02.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:02.015+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:02.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:02.431+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:02.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:02.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:02.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:02.703+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:03.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:03.603+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:03.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:03.707+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:03.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:03.807+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376947b00 0x5623756c6000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.024 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:04.019+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926d28680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.025 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:04.020+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:04.211+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.227 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:04.223+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.228 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:04.220+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.631 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:04.627+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830cd80 0x55d926ec4680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:04.628+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:04.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:04.703+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:05.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:05.015+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:05.436 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:05.428+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:05.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:05.431+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:05.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:05.731+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:06.608 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:06.603+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:06.609 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:06.604+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576585200 0x556576346680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:06.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:06.687+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:06.812 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:06.807+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926d28680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:06.813 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:06.808+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:07.019+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.025 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:07.019+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.217 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:07.212+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.217 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:07.211+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830d200 0x55d926f64680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:07.223+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376947680 0x5623756c6b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.228 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:07.223+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:07.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:07.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:07.627+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948400 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.632 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:07.627+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:07.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:07.679+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:08.020 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:08.016+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:08.062 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:08.015+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:08.435 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:08.431+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d928318000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:08.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:08.431+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:08.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:08.643+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:09.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:09.599+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:09.609 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:09.604+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576585f80 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:09.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:09.603+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:09.610 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:09.603+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830d680 0x55d928318580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:09.812 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:09.808+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:09.813 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:09.807+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:09.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:09.807+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:10.024 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:10.020+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:10.216 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:10.211+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:10.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:10.211+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948400 0x5623756c6b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:10.217 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:10.212+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13a80 0x556576a12000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:10.227 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:10.220+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6db00 0x556576cbdb80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:10.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:10.603+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:10.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:10.624+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576586880 0x556576cbf180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:11.020 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:11.016+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b10d80 0x556576a12b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:11.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:11.015+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:11.021 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:11.015+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:11.435 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:11.428+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:11.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:11.631+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:12.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:12.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:12.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:12.611+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:13.025 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:13.019+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948d00 0x5623756c7080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.025 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:13.019+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830db00 0x55d928318b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.026 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:13.019+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:13.223+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.228 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:13.223+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.228 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:13.223+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:13.599+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:13.631 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:13.627+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d928318000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:13.627+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:13.632 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:13.627+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830df80 0x55d928319080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:14.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:14.431+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:14.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:14.431+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:14.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:14.431+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:14.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:14.647+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:15.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:15.623+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:16.024 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:16.020+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:16.040 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:16.035+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:16.631 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:16.624+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:16.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:16.643+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:17.436 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:17.428+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:17.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:17.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:17.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:17.615+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:18.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:18.655+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:19.244 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:19.239+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:19.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:19.687+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:20.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:20.663+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:21.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:21.671+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:22.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:22.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:22.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:22.691+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:23.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:23.019+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:23.027 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:23.020+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c480 0x556576a15180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:23.228 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:23.224+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:23.229 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:23.223+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:23.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:23.628+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:23.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:23.627+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376949180 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:23.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:23.679+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:24.438 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:24.432+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c900 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:24.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:24.431+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:24.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:24.651+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:25.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:25.619+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:26.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:26.019+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:26.027 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:26.020+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:26.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:26.223+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:26.228 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:26.224+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c000 0x556576a13600 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:26.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:26.615+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:26.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:26.627+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376949180 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:26.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:26.628+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:27.436 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:27.432+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:27.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:27.431+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:27.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:27.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:27.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:27.591+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:28.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:28.619+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:29.027 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:29.023+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:29.228 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:29.223+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:29.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:29.591+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:29.632 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:29.627+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d928318000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:30.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:30.431+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830e880 0x55d928319b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:30.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:30.591+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:31.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:31.551+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:32.027 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:32.020+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:32.027 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:32.023+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:32.229 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:32.224+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:32.229 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:32.223+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:32.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:32.503+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:32.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:32.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:32.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:32.628+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:32.633 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:32.627+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:33.436 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:33.432+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c900 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:33.437 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:33.431+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d928318000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:33.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:33.551+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:34.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:34.583+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:35.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:35.022+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:35.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:35.222+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:35.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:35.618+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:35.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:35.626+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376949180 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:36.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:36.430+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:36.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:36.646+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:37.595 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:37.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:37.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:37.618+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:38.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:38.022+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:38.028 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:38.024+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:38.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:38.226+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:38.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:38.630+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376949180 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:38.637 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:38.632+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:38.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:38.650+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:39.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:39.434+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:39.441 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:39.436+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:39.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:39.618+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:40.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:40.594+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:41.028 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:41.022+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:41.232 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:41.226+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:41.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:41.630+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:42.440 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:42.434+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d928318000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:42.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:42.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:42.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:42.666+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:43.026 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:43.020+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:43.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:43.018+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:43.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:43.222+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:43.229 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:43.224+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:43.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:43.628+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:43.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:43.626+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:43.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:43.634+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:44.436 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:44.432+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c900 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:44.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:44.430+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376949180 0x5623759e5600 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:44.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:44.654+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:45.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:45.614+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:46.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:46.034+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:46.041 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:46.036+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:46.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:46.654+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:47.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:47.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:47.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:47.626+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:48.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:48.638+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:49.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:49.238+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:49.245 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:49.239+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6f180 0x556576a14680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:49.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:49.618+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:50.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:50.614+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:51.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:51.614+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:52.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:52.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:52.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:52.598+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:53.027 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:53.018+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830f180 0x55d92831b180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:53.230 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:53.222+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:53.632 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:53.626+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:53.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:53.634+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:54.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:54.430+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:54.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:54.594+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:55.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:55.642+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:56.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:56.018+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376948880 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:56.027 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:56.019+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:56.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:56.222+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:56.228 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:56.223+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:56.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:56.602+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:56.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:56.626+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:56.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:56.627+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c900 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:57.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:57.430+0000 7fc49c8a7700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:57.437 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:56:57.431+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:57.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:56:57.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:56:57.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:57.650+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:58.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:58.642+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:56:59.027 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:59.022+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d92831b700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:59.228 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:59.222+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830f600 0x55d928318580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:59.632 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:56:59.626+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:56:59.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:56:59.634+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:00.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:00.430+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec5700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:00.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:00.658+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:01.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:01.622+0000 7fd46a48c700 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:02.027 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:02.019+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a13080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:02.028 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:02.022+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:02.228 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:02.223+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576b13600 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:02.228 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:02.222+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d92831b700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:02.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:02.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:02.632 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:02.627+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c900 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:02.633 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:02.626+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830fa80 0x55d928319600 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:02.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:02.670+0000 7fd46a48c700 -1 osd.1 27 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:03.436 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:03.430+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228000 0x55d926f65700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:03.437 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:03.431+0000 7f8a0fb1d700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:03.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:03.682+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:04.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:04.694+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:05.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:05.022+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562376949600 0x5623759e4580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:05.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:05.226+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:05.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:05.630+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:05.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:05.718+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:06.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:06.434+0000 7fc49d0a8700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:06.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:06.678+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:07.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:07.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:07.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:07.706+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:08.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:08.226+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375873f80 0x5623756c9700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:08.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:08.630+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375875180 0x5623756c8680 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:08.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:08.678+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:09.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:09.433+0000 7fc49789d700 -1 --2- [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] >> conn(0x562375a36400 0x5623756c9180 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:09.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:09.653+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:10.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:10.625+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:11.028 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:11.021+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927228900 0x55d926ec3080 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:11.029 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:11.023+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6c900 0x556576cbf700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:11.232 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:11.227+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576d6cd80 0x556576cbc000 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:11.232 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:11.225+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d927229680 0x55d92831b700 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:11.636 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:11.629+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92722ad00 0x55d928319b80 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:11.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:11.629+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:12.440 INFO:tasks.ceph.mon.b.smithi204.stderr:2019-10-17T20:57:12.435+0000 7f8a0f31c700 -1 --2- [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] >> conn(0x556576587600 0x556576a12580 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:12.441 INFO:tasks.ceph.mon.c.smithi092.stderr:2019-10-17T20:57:12.433+0000 7fce7d783700 -1 --2- [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] >> conn(0x55d92830c900 0x55d928318b00 secure :-1 s=AUTH_ACCEPTING_MORE pgs=0 cs=0 l=1 rx=0 tx=0).run_continuation failed decoding of frame header: buffer::end_of_buffer 2019-10-17T20:57:12.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:12.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:12.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:12.665+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:13.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:13.657+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:14.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:14.649+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:15.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:15.637+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:16.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:16.677+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:17.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:17.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:17.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:17.661+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:18.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:18.657+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:19.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:19.677+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:20.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:20.717+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:21.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:21.749+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:22.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:22.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:22.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:22.733+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:23.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:23.693+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:24.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:24.689+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:25.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:25.705+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:26.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:26.681+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:27.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:27.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:27.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:27.633+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:28.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:28.625+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:29.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:29.673+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:30.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:30.641+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:31.597+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:32.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:32.577+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:32.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:32.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:33.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:33.545+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:34.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:34.553+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:35.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:35.505+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:36.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:36.465+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:37.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:37.465+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:37.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:37.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:38.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:38.437+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:39.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:39.397+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:40.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:40.437+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:41.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:41.449+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:42.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:42.421+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:42.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:42.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:43.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:43.457+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:44.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:44.416+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:45.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:45.436+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:46.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:46.484+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:47.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:47.460+0000 7fd46a48c700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:47.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:47.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:48.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:48.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:49.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:49.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:50.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:50.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:51.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:51.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:52.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:52.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:52.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:52.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:53.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:53.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:54.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:54.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:55.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:55.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:56.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:56.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:57.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:57.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:57.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:57:57.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:57:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:58.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:57:59.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:57:59.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:00.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:00.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:01.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:01.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:02.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:02.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:02.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:02.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:03.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:03.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:04.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:04.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:05.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:05.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:06.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:06.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:07.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:07.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:07.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:07.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:08.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:08.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:09.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:09.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:10.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:10.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:11.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:11.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:12.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:12.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:12.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:12.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:13.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:13.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:14.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:14.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:15.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:15.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:16.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:16.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:17.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:17.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:17.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:17.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:18.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:18.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:18.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:18.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:19.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:19.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:19.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:19.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:20.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:20.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:20.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:20.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:21.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:21.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:21.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:21.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:22.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:22.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:22.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:22.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:22.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:22.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:23.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:23.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:23.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:23.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:24.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:24.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:24.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:24.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:25.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:25.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:25.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:25.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:26.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:26.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:26.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:27.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:27.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:27.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:27.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:27.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:27.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:28.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:28.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:28.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:28.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:29.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:29.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:29.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:29.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:30.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:30.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:30.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:30.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:31.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:31.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:31.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:31.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:32.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:32.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:32.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:32.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:32.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:32.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:33.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:33.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:33.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:33.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:34.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:34.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:34.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:34.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:35.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:35.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:35.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:35.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:36.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:36.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:36.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:36.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:37.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:37.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:37.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:37.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:37.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:37.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:38.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:38.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:38.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:38.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:39.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:39.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:39.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:39.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:40.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:40.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:40.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:40.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:41.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:41.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:41.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:41.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:42.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:42.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:42.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:42.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:42.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:42.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:43.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:43.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:43.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:43.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:44.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:44.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:44.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:44.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:45.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:45.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:45.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:45.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:46.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:46.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:46.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:46.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:47.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:47.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:47.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:47.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:47.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:47.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:48.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:48.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:48.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:48.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:49.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:49.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:49.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:49.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:50.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:50.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:50.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:50.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:51.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:51.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:51.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:51.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:52.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:52.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:52.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:52.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:52.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:52.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:53.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:53.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:53.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:53.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:54.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:54.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:54.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:54.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:55.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:55.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:55.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:55.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:56.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:56.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:56.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:56.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:57.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:57.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:57.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:57.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:57.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:58:57.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:58:58.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:58.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:58.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:58.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:58:59.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:59.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:58:59.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:58:59.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:00.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:00.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:00.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:00.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:01.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:01.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:01.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:01.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:02.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:02.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:02.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:02.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:02.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:02.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:03.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:03.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:03.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:03.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:04.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:04.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:04.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:04.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:05.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:05.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:05.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:05.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:06.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:06.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:06.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:06.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:07.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:07.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:07.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:07.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:07.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:07.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:08.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:08.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:08.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:08.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:09.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:09.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:09.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:09.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:10.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:10.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:10.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:10.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:11.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:11.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:11.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:11.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:12.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:12.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:12.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:12.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:13.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:13.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:13.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:13.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:14.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:14.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:14.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:14.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:15.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:15.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:15.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:15.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:16.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:16.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:16.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:16.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:17.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:17.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:17.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:17.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:17.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:17.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:18.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:18.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:18.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:18.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:19.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:19.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:19.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:19.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:20.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:20.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:20.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:20.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:21.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:21.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:21.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:21.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:22.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:22.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:22.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:22.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:22.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:22.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:23.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:23.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:23.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:23.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:24.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:24.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:24.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:24.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:25.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:25.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:25.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:25.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:26.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:26.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:26.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:26.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:27.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:27.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:27.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:27.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:27.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:27.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:28.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:28.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:28.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:28.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:29.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:29.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:29.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:29.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:30.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:30.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:30.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:30.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:31.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:31.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:31.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:31.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:32.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:32.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:32.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:32.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:32.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:32.613+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:33.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:33.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:33.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:33.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:34.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:34.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:34.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:34.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:35.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:34.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:35.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:34.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:36.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:36.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:36.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:36.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:37.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:37.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:37.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:37.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:37.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:37.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:37.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:37.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:37.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:37.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:39.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:39.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:39.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:39.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:40.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:40.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:40.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:41.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:41.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:41.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:41.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:42.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:42.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:42.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:42.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:42.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:42.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:43.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:43.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:43.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:43.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:44.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:44.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:44.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:44.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:45.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:45.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:45.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:45.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:46.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:46.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:46.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:46.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:47.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:47.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:47.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:47.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:47.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:47.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:48.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:48.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:48.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:48.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:49.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:49.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:49.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:49.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:50.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:50.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:50.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:50.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:51.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:51.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:51.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:51.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:52.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:52.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:52.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:52.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:52.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:52.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:53.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:53.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:53.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:53.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:54.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:54.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:54.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:54.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:55.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:55.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:55.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:55.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:56.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:56.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:56.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:56.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:57.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:57.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:57.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:57.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:57.626 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T20:59:57.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T20:59:58.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:58.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:58.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:58.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T20:59:59.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:59.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T20:59:59.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T20:59:59.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:00.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:00.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:00.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:00.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:01.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:01.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:01.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:01.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:02.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:02.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:02.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:02.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:02.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:02.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:03.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:03.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:03.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:03.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:04.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:04.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:04.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:04.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:05.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:05.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:05.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:05.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:06.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:06.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:06.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:06.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:07.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:07.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:07.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:07.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:07.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:07.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:08.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:08.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:08.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:08.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:09.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:09.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:09.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:09.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:10.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:10.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:10.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:10.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:11.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:11.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:11.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:11.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:12.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:12.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:12.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:12.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:12.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:12.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:13.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:13.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:13.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:13.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:14.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:14.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:14.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:14.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:15.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:15.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:15.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:15.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:16.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:16.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:16.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:16.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:17.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:17.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:17.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:17.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:17.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:17.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:18.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:18.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:19.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:19.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:19.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:19.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:20.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:20.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:20.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:20.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:21.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:21.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:21.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:21.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:22.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:22.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:22.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:22.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:22.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:22.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:23.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:23.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:23.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:23.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:24.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:24.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:25.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:25.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:25.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:25.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:26.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:26.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:26.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:26.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:27.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:27.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:27.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:27.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:27.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:27.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:28.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:28.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:28.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:28.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:29.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:29.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:29.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:29.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:30.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:30.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:30.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:30.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:31.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:31.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:31.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:31.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:31.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:31.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:31.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:31.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:32.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:32.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:32.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:32.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:32.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:32.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:33.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:33.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:33.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:33.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:34.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:34.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:34.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:34.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:35.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:35.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:35.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:35.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:37.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:37.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:37.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:37.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:37.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:37.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:38.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:38.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:38.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:38.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:39.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:39.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:39.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:39.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:40.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:40.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:40.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:40.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:41.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:41.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:42.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:42.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:42.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:42.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:42.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:42.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:43.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:43.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:43.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:43.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:43.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:43.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:43.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:43.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:44.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:44.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:45.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:45.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:45.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:45.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:46.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:46.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:46.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:46.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:47.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:47.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:47.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:47.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:47.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:47.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:48.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:48.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:48.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:48.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:50.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:50.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:50.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:50.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:50.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:50.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:50.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:50.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:52.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:52.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:52.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:52.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:52.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:52.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:53.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:53.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:53.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:53.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:54.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:54.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:54.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:54.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:55.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:55.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:55.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:55.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:55.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:55.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:55.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:55.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:56.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:56.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 since back 2019-10-17T20:47:58.841659+0000 front 2019-10-17T20:47:58.841568+0000 (oldest deadline 2019-10-17T20:50:39.966758+0000) 2019-10-17T21:00:56.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:56.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:57.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:00:57.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:00:57.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:57.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:00:59.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:00:59.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:00.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:00.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:01.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:01.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:02.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:02.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:02.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:02.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:03.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:03.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:04.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:05.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:05.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:06.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:06.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:07.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:07.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:07.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:07.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:08.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:08.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:09.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:08.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:09.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:09.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:10.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:10.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:11.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:11.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:12.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:12.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:12.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:12.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:13.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:13.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:14.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:14.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:15.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:15.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:16.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:16.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:17.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:17.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:17.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:17.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:18.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:18.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:19.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:19.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:19.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:19.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:20.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:20.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:20.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:20.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:21.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:21.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:21.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:21.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:22.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:22.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:22.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:22.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:22.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:22.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:23.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:23.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:23.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:23.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:24.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:24.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:24.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:24.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:25.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:25.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:25.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:25.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:26.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:26.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:26.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:26.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:27.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:27.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:27.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:27.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:28.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:28.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:28.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:28.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:29.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:29.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:29.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:29.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:31.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:31.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:31.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:31.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:32.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:32.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:32.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:32.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:32.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:32.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:33.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:33.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:33.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:33.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:34.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:34.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:34.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:34.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:35.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:35.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:35.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:35.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:36.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:36.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:36.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:36.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:37.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:37.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:37.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:37.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:37.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:37.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:37.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:37.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:37.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:37.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:39.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:39.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:39.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:39.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:40.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:40.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:40.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:40.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:41.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:41.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:41.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:41.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:42.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:42.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:42.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:42.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:42.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:42.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:43.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:43.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:43.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:43.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:44.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:44.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:44.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:44.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:45.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:45.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:45.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:45.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:46.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:46.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:46.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:46.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:47.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:47.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:47.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:47.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:47.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:47.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:48.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:48.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:48.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:48.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:49.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:49.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:49.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:49.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:50.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:50.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:50.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:50.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:51.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:51.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:51.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:51.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:52.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:52.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:52.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:52.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:52.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:52.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:53.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:53.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:53.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:53.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:54.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:54.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:54.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:54.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:55.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:55.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:55.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:55.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:56.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:56.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:56.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:56.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:57.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:57.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:57.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:57.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:57.644 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:01:57.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:01:58.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:58.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:58.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:58.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:01:59.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:59.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:01:59.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:01:59.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:00.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:00.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:00.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:00.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:01.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:01.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:01.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:01.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:02.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:02.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:02.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:02.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:02.644 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:02.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:03.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:03.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:03.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:03.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:03.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:03.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:03.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:03.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:05.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:04.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:05.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:04.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:06.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:06.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:06.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:06.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:07.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:06.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:07.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:06.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:07.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:07.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:08.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:08.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:08.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:08.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:08.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:08.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:08.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:08.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:10.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:10.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:10.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:10.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:11.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:10.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:11.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:10.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:11.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:11.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:11.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:11.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:12.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:12.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:13.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:13.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:13.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:13.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:13.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:13.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:13.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:13.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:14.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:14.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:14.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:14.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:15.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:15.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:15.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:15.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:16.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:16.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:16.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:16.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:17.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:17.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:17.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:17.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:17.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:17.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:18.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:18.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:18.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:18.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:19.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:19.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:19.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:19.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:20.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:20.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:20.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:20.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:22.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:22.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:22.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:22.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:22.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:22.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:23.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:23.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:23.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:23.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:24.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:24.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:24.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:24.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:24.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:24.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:24.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:24.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:26.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:26.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:26.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:26.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:27.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:27.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:27.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:27.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:27.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:27.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:28.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:28.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:28.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:28.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:29.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:29.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:29.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:30.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:30.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:30.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:30.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:30.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:30.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:30.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:30.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:32.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:32.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:32.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:32.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:32.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:32.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:33.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:33.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:33.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:33.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:34.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:34.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:34.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:34.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:35.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:35.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:35.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:35.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:36.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:36.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:36.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:36.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:37.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:37.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:37.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:37.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:37.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:37.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:38.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:38.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:38.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:38.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:39.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:39.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:39.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:39.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:40.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:40.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:40.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:40.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:41.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:41.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:42.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:42.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:42.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:42.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:42.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:42.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:43.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:43.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:43.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:43.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:44.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:44.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:44.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:44.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:45.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:45.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:45.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:45.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:46.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:46.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:46.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:46.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:47.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:47.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:47.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:47.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:47.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:47.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:48.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:48.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:48.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:48.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:49.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:49.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:49.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:49.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:50.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:50.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:50.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:50.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:51.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:51.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:51.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:51.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:52.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:52.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:52.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:52.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:52.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:52.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:53.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:53.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:54.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:54.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:54.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:54.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:55.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:55.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:55.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:55.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:56.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:56.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:56.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:56.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:57.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:57.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:57.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:57.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:57.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:02:57.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:02:58.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:58.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:58.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:58.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:02:59.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:59.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:02:59.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:02:59.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:00.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:00.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:00.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:00.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:01.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:01.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:01.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:01.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:02.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:02.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:02.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:02.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:02.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:02.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:03.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:03.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:03.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:03.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:04.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:04.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:04.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:04.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:05.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:05.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:05.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:05.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:06.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:06.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:06.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:06.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:07.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:07.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:07.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:07.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:07.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:07.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:08.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:08.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:08.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:08.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:09.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:09.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:09.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:09.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:10.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:10.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:10.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:10.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:11.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:11.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:11.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:11.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:12.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:12.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:12.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:12.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:12.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:12.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:13.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:13.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:13.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:13.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:14.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:14.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:14.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:14.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:15.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:15.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:15.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:15.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:16.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:16.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:16.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:16.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:17.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:17.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:17.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:17.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:17.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:17.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:18.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:18.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:18.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:18.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:19.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:19.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:19.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:19.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:20.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:20.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:20.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:20.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:21.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:21.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:21.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:21.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:22.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:22.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:22.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:22.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:22.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:22.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:23.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:23.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:23.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:23.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:24.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:24.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:25.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:25.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:25.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:25.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:26.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:26.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:26.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:26.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:27.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:27.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:27.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:27.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:27.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:27.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:28.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:28.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:28.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:28.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:29.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:29.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:29.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:29.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:30.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:30.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:30.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:30.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:31.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:31.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:31.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:31.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:32.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:32.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:32.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:32.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:32.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:32.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:33.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:33.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:33.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:33.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:34.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:34.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:34.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:34.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:35.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:35.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:35.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:35.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:36.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:36.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:36.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:36.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:37.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:37.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:37.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:37.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:37.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:37.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:38.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:38.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:38.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:38.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:39.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:39.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:39.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:39.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:40.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:40.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:40.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:40.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:41.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:41.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:41.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:41.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:42.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:42.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:42.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:42.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:42.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:42.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:43.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:43.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:43.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:43.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:44.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:44.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:44.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:44.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:45.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:45.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:45.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:45.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:46.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:46.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:46.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:46.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:47.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:47.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:47.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:47.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:47.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:47.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:48.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:48.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:48.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:48.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:49.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:49.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:50.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:50.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:50.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:50.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:51.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:51.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:51.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:51.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:52.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:52.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:52.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:52.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:52.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:52.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:53.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:53.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:53.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:53.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:54.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:54.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:54.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:54.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:55.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:55.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:55.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:55.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:56.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:56.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:56.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:56.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:57.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:57.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:57.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:57.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:57.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:03:57.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:03:58.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:58.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:58.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:58.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:03:59.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:59.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:03:59.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:03:59.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:00.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:00.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:00.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:00.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:01.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:01.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:01.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:01.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:02.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:02.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:02.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:02.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:02.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:02.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:03.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:03.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:03.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:03.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:04.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:04.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:04.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:04.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:05.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:05.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:05.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:05.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:06.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:06.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:06.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:06.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:07.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:07.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:07.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:07.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:07.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:07.658+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:08.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:08.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:08.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:08.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:09.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:09.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:09.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:09.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:10.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:10.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:10.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:10.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:11.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:11.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:11.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:11.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:12.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:12.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:12.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:12.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:12.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:12.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:13.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:13.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:13.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:13.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:14.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:14.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:14.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:14.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:15.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:15.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:15.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:15.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:16.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:16.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:16.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:16.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:17.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:17.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:17.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:17.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:17.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:17.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:18.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:18.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:18.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:18.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:19.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:19.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:19.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:19.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:20.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:20.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:20.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:20.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:21.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:21.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:21.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:21.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:22.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:22.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:22.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:22.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:22.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:22.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:23.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:23.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:23.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:23.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:24.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:24.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:24.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:24.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:25.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:25.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:26.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:26.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:26.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:26.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:27.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:27.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:27.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:27.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:27.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:27.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:28.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:28.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:28.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:28.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:29.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:29.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:29.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:29.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:30.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:30.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:30.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:30.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:31.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:31.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:31.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:31.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:32.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:32.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:32.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:32.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:32.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:32.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:33.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:33.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:33.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:33.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:34.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:34.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:34.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:34.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:35.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:35.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:35.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:35.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:36.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:36.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:36.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:36.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:37.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:37.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:37.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:37.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:37.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:37.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:38.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:38.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:38.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:38.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:39.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:39.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:39.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:39.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:40.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:40.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:41.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:41.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:42.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:42.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:42.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:42.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:42.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:42.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:43.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:43.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:43.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:43.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:44.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:44.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:44.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:44.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:45.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:45.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:45.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:45.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:46.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:46.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:46.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:46.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:47.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:47.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:47.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:47.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:47.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:47.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:48.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:48.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:48.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:48.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:49.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:49.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:49.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:49.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:50.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:50.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:50.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:50.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:51.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:51.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:51.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:51.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:52.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:52.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:52.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:52.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:52.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:52.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:53.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:53.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:53.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:53.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:54.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:54.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:54.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:54.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:55.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:55.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:55.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:55.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:56.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:56.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:56.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:56.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:57.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:57.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:57.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:57.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:57.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:04:57.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:04:58.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:58.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:58.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:58.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:04:59.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:59.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:04:59.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:04:59.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:00.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:00.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:00.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:00.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:01.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:01.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:01.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:01.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:02.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:02.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:02.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:02.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:02.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:02.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:03.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:03.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:03.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:03.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:04.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:04.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:04.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:04.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:05.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:05.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:05.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:05.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:06.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:06.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:06.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:06.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:07.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:07.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:07.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:07.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:07.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:07.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:08.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:08.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:08.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:08.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:09.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:09.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:09.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:09.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:10.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:10.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:10.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:10.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:11.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:11.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:11.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:11.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:12.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:12.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:12.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:12.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:12.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:12.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:13.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:13.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:13.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:13.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:14.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:14.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:14.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:14.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:15.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:15.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:15.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:15.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:16.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:16.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:16.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:16.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:17.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:17.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:17.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:17.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:17.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:17.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:18.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:18.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:18.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:18.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:19.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:19.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:19.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:19.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:20.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:20.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:20.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:20.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:21.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:21.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:21.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:21.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:22.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:22.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:22.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:22.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:22.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:22.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:23.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:23.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:23.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:23.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:24.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:24.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:24.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:24.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:25.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:25.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:25.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:25.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:26.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:26.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:27.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:27.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:27.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:27.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:27.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:27.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:28.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:28.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:28.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:28.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:29.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:29.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:29.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:29.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:30.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:30.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:30.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:30.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:31.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:31.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:31.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:31.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:32.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:32.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:32.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:32.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:32.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:33.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:33.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:33.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:33.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:34.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:34.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:34.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:34.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:35.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:35.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:35.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:35.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:36.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:36.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:36.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:36.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:37.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:37.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:37.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:37.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:37.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:37.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:38.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:38.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:39.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:39.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:39.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:39.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:40.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:40.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:40.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:40.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:41.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:41.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:41.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:41.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:42.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:42.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:42.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:42.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:42.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:42.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:43.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:43.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:43.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:43.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:44.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:44.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:44.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:44.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:45.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:45.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:45.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:45.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:46.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:46.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:46.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:46.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:47.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:47.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:47.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:47.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:47.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:47.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:48.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:48.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:48.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:48.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:49.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:49.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:49.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:49.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:50.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:50.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:50.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:50.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:51.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:51.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:51.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:51.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:52.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:52.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:52.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:52.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:52.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:52.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:53.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:53.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:53.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:53.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:54.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:54.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:54.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:54.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:55.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:55.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:55.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:55.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:56.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:56.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:57.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:57.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:57.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:57.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:57.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:05:57.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:05:58.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:58.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:58.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:58.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:05:59.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:59.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:05:59.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:05:59.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:00.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:00.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:00.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:00.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:01.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:01.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:01.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:01.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:02.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:02.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:02.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:02.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:02.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:02.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:03.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:03.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:03.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:03.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:04.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:04.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:04.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:04.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:05.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:05.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:05.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:05.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:06.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:06.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:06.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:06.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:07.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:07.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:07.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:07.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:07.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:07.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:08.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:08.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:08.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:08.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:09.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:09.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:09.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:09.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:10.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:10.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:10.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:10.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:11.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:11.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:11.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:11.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:12.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:12.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:12.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:12.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:12.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:12.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:13.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:13.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:13.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:13.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:14.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:14.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:14.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:14.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:15.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:15.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:15.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:15.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:16.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:16.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:16.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:16.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:17.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:17.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:17.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:17.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:17.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:17.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:18.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:18.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:18.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:18.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:19.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:19.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:19.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:19.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:20.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:20.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:20.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:20.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:21.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:21.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:21.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:21.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:22.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:22.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:22.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:22.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:22.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:22.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:23.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:23.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:23.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:23.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:24.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:24.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:24.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:24.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:25.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:25.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:26.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:26.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:26.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:26.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:27.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:27.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:27.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:27.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:27.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:27.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:28.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:28.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:28.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:28.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:29.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:29.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:29.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:29.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:30.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:30.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:30.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:30.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:31.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:31.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:31.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:31.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:32.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:32.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:32.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:32.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:32.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:32.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:33.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:33.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:33.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:33.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:34.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:34.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:34.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:34.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:35.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:35.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:35.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:35.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:36.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:36.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:36.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:36.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:37.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:37.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:37.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:37.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:37.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:37.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:38.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:38.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:38.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:38.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:39.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:39.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:39.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:39.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:40.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:40.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:40.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:40.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:41.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:41.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:41.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:41.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:42.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:42.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:42.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:42.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:42.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:42.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:43.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:43.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:43.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:43.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:44.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:44.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:44.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:44.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:45.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:45.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:45.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:45.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:46.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:46.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:46.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:46.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:47.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:47.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:47.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:47.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:47.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:47.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:48.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:48.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:48.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:48.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:49.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:49.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:49.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:49.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:49.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:49.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:49.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:49.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:50.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:50.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:50.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:50.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:51.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:51.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:51.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:51.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:52.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:52.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:52.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:52.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:52.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:52.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:53.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:53.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:53.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:53.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:54.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:54.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:54.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:54.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:55.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:55.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:55.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:55.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:56.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:56.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:56.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:56.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:57.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:06:57.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:06:57.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:57.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:57.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:57.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:58.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:58.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:58.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:58.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:06:59.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:59.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:06:59.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:06:59.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:00.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:00.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:00.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:00.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:01.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:01.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:01.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:01.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:02.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:02.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:02.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:02.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:02.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:02.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:03.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:03.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:03.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:03.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:04.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:04.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:04.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:04.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:05.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:05.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:05.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:05.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:06.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:06.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:06.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:06.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:07.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:07.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:07.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:07.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:07.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:07.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:08.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:08.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:08.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:08.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:09.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:09.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:09.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:09.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:10.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:10.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:10.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:10.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:11.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:11.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:11.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:11.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:12.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:12.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:12.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:12.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:12.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:12.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:13.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:13.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:13.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:13.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:14.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:14.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:14.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:14.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:15.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:15.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:15.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:15.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:16.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:16.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:16.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:16.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:17.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:17.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:17.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:17.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:17.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:17.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:18.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:18.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:18.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:18.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:19.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:19.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:19.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:19.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:20.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:20.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:20.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:20.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:21.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:21.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:21.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:21.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:22.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:22.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:22.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:22.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:22.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:22.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:23.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:23.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:23.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:23.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:24.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:24.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:25.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:25.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:25.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:25.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:26.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:26.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:26.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:26.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:27.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:27.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:27.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:27.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:27.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:27.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:28.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:28.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:28.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:28.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:29.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:29.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:30.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:30.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:30.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:30.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:31.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:31.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:31.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:31.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:32.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:32.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:32.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:32.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:32.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:32.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:33.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:33.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:33.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:33.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:34.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:34.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:34.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:34.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:35.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:35.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:35.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:35.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:36.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:36.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:36.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:36.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:37.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:37.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:37.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:37.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:37.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:37.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:38.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:38.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:38.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:38.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:39.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:39.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:39.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:39.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:40.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:40.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:40.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:40.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:41.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:41.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:41.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:41.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:42.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:42.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:42.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:42.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:42.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:42.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:43.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:43.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:43.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:43.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:44.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:44.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:44.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:44.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:45.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:45.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:45.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:45.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:46.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:46.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:46.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:46.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:47.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:47.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:47.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:47.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:47.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:47.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:48.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:48.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:48.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:48.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:49.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:49.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:49.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:49.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:50.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:50.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:50.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:50.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:51.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:51.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:51.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:51.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:52.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:52.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:52.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:52.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:52.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:52.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:53.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:53.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:53.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:53.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:54.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:54.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:54.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:54.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:55.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:55.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:55.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:55.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:56.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:56.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:56.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:56.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:57.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:07:57.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:07:57.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:57.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:57.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:57.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:58.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:58.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:58.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:58.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:07:59.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:59.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:07:59.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:07:59.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:00.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:00.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:00.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:00.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:01.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:01.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:01.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:01.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:02.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:02.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:02.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:02.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:02.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:02.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:03.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:03.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:03.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:03.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:04.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:04.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:04.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:04.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:05.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:05.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:05.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:05.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:06.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:06.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:06.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:06.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:07.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:07.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:07.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:07.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:07.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:07.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:08.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:08.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:08.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:08.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:09.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:09.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:09.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:09.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:10.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:10.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:10.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:10.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:11.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:11.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:11.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:11.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:12.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:12.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:12.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:12.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:12.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:12.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:13.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:13.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:13.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:13.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:14.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:14.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:14.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:14.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:15.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:15.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:15.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:15.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:16.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:16.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:16.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:16.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:17.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:17.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:17.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:17.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:17.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:17.694+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:18.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:18.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:18.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:18.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:19.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:19.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:19.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:19.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:20.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:20.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:20.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:20.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:21.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:21.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:21.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:21.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:22.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:22.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:22.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:22.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:22.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:22.694+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:23.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:23.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:23.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:23.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:24.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:24.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:24.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:24.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:25.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:25.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:25.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:25.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:26.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:26.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:26.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:26.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:27.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:27.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:27.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:27.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:27.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:27.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:28.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:28.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:28.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:28.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:29.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:29.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:29.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:29.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:30.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:30.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:31.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:31.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:31.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:31.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:32.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:32.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:32.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:32.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:32.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:32.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:33.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:33.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:33.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:33.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:34.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:34.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:34.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:34.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:35.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:35.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:35.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:35.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:36.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:36.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:36.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:36.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:37.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:37.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:37.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:37.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:37.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:37.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:38.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:38.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:38.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:38.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:39.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:39.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:39.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:39.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:40.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:40.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:40.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:40.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:41.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:41.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:41.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:41.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:42.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:42.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:42.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:42.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:42.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:42.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:43.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:43.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:43.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:43.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:44.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:44.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:44.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:44.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:45.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:45.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:45.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:45.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:46.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:46.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:46.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:46.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:47.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:47.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:47.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:47.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:47.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:47.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:48.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:48.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:48.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:48.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:49.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:49.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:49.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:49.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:50.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:50.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:50.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:50.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:51.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:51.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:51.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:51.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:52.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:52.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:52.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:52.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:52.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:52.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:53.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:53.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:53.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:53.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:54.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:54.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:54.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:54.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:55.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:55.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:55.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:55.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:56.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:56.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:56.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:56.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:57.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:57.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:57.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:57.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:57.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:08:57.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:08:58.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:58.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:58.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:58.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:08:59.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:59.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:08:59.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:08:59.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:00.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:00.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:00.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:00.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:01.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:01.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:01.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:01.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:02.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:02.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:02.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:02.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:02.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:02.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:03.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:03.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:03.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:03.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:04.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:04.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:04.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:04.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:05.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:05.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:05.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:05.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:06.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:06.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:06.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:06.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:07.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:07.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:07.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:07.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:07.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:07.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:08.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:08.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:08.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:08.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:09.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:09.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:09.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:09.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:09.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:09.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:09.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:09.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:11.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:11.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:11.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:11.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:11.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:11.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:11.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:11.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:12.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:12.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:13.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:13.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:13.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:13.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:14.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:14.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:14.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:14.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:14.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:14.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:14.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:14.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:15.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:15.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:15.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:15.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:17.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:17.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:17.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:17.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:17.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:17.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:18.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:18.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:18.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:18.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:19.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:19.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:19.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:19.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:20.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:20.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:20.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:20.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:21.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:21.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:21.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:21.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:22.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:22.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:22.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:22.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:22.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:22.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:23.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:23.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:23.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:23.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:24.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:24.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:24.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:24.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:25.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:25.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:25.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:25.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:26.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:26.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:26.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:26.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:27.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:27.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:27.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:27.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:27.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:27.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:28.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:28.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:28.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:28.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:29.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:29.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:29.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:29.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:30.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:30.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:30.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:30.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:31.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:31.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:31.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:31.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:32.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:32.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:32.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:32.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:32.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:32.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:33.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:33.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:33.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:33.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:34.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:34.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:34.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:34.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:35.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:35.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:35.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:35.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:36.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:36.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:36.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:36.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:37.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:37.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:37.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:37.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:37.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:37.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:38.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:38.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:38.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:38.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:39.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:39.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:39.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:39.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:40.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:40.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:40.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:40.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:41.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:41.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:42.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:42.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:42.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:42.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:42.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:42.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:43.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:43.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:43.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:43.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:44.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:44.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:44.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:44.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:45.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:45.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:45.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:45.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:46.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:46.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:46.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:46.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:47.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:47.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:47.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:47.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:47.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:47.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:48.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:48.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:48.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:48.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:49.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:49.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:49.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:49.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:50.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:50.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:50.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:50.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:51.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:51.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:51.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:51.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:52.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:52.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:52.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:52.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:52.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:52.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:53.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:53.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:53.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:53.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:54.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:54.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:54.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:54.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:55.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:55.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:55.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:55.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:56.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:56.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:56.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:56.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:57.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:57.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:57.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:57.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:57.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:09:57.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:09:58.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:58.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:58.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:58.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:09:59.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:59.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:09:59.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:09:59.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:00.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:00.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:00.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:00.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:01.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:01.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:01.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:01.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:02.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:02.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:02.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:02.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:03.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:03.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:03.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:03.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:04.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:04.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:04.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:04.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:05.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:05.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:05.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:05.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:06.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:06.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:06.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:06.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:07.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:07.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:07.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:07.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:07.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:07.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:08.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:08.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:08.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:08.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:09.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:09.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:09.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:09.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:10.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:10.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:10.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:10.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:11.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:11.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:11.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:11.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:12.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:12.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:12.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:12.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:12.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:12.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:13.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:13.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:13.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:13.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:14.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:14.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:14.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:14.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:15.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:15.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:15.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:15.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:16.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:16.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:16.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:16.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:17.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:17.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:17.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:17.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:17.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:17.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:18.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:18.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:18.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:18.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:19.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:19.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:19.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:19.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:20.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:20.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:20.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:20.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:21.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:21.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:21.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:21.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:22.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:22.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:22.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:22.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:22.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:22.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:23.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:23.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:23.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:23.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:24.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:24.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:24.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:24.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:25.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:25.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:25.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:25.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:26.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:26.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:26.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:26.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:27.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:27.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:27.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:27.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:27.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:27.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:28.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:28.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:28.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:28.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:29.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:29.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:29.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:29.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:30.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:30.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:30.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:30.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:31.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:31.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:31.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:31.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:32.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:32.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:32.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:32.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:32.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:32.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:33.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:33.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:33.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:33.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:34.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:34.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:34.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:34.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:35.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:35.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:35.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:35.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:36.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:36.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:36.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:36.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:37.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:37.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:37.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:37.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:37.723 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:37.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:38.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:38.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:38.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:38.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:39.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:39.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:39.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:39.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:40.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:40.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:41.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:41.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:41.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:41.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:42.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:42.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:42.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:42.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:43.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:43.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:43.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:43.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:44.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:44.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:44.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:44.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:45.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:45.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:45.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:45.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:46.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:46.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:46.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:46.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:47.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:47.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:47.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:47.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:47.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:47.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:48.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:48.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:48.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:48.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:49.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:49.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:49.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:49.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:50.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:50.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:50.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:50.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:51.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:51.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:51.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:51.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:52.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:52.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:52.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:52.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:52.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:52.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:53.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:53.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:53.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:53.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:54.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:54.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:54.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:54.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:55.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:55.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:55.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:55.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:56.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:56.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:56.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:56.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:57.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:57.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:57.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:57.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:57.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:10:57.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:10:58.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:58.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:58.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:58.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:10:59.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:59.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:10:59.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:10:59.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:00.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:00.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:00.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:00.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:01.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:01.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:01.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:01.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:02.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:02.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:02.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:02.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:02.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:02.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:03.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:03.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:03.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:03.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:04.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:04.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:04.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:04.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:05.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:05.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:05.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:05.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:06.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:06.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:06.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:06.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:07.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:07.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:07.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:07.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:07.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:07.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:08.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:08.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:08.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:08.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:09.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:09.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:09.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:09.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:10.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:10.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:10.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:10.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:11.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:11.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:11.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:12.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:12.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:12.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:12.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:12.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:12.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:13.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:13.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:13.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:13.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:14.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:14.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:14.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:14.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:15.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:15.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:15.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:15.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:16.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:16.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:16.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:16.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:17.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:17.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:17.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:17.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:17.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:17.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:18.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:18.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:19.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:19.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:19.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:19.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:20.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:20.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:20.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:21.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:21.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:21.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:21.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:22.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:22.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:22.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:22.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:22.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:22.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:23.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:23.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:23.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:23.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:24.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:24.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:24.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:24.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:25.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:25.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:25.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:25.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:26.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:26.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:26.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:27.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:27.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:27.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:27.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:27.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:27.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:28.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:28.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:28.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:28.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:29.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:29.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:00:59.693507+0000 (oldest deadline 2019-10-17T21:01:19.693507+0000) 2019-10-17T21:11:29.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:29.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:30.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:30.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:31.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:31.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:32.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:32.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:32.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:33.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:33.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:34.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:34.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:35.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:35.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:36.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:36.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:37.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:37.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:37.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:37.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:38.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:38.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:39.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:39.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:40.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:40.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:41.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:41.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:42.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:42.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:42.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:42.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:43.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:43.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:44.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:44.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:45.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:45.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:46.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:46.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:47.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:47.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:47.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:47.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:48.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:48.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:49.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:49.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:50.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:50.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:50.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:50.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:51.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:51.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:51.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:51.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:52.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:52.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:52.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:52.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:52.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:53.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:53.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:53.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:53.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:54.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:54.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:54.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:54.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:55.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:55.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:55.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:56.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:56.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:56.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:56.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:57.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:57.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:57.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:57.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:57.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:11:57.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:11:58.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:58.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:58.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:58.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:11:59.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:59.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:11:59.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:11:59.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:00.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:00.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:00.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:01.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:01.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:01.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:01.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:02.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:02.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:02.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:02.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:02.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:02.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:03.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:03.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:03.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:03.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:04.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:04.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:05.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:05.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:05.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:05.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:06.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:06.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:06.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:06.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:07.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:07.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:07.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:07.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:07.737 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:07.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:08.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:08.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:08.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:08.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:09.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:09.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:09.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:09.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:10.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:10.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:10.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:10.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:11.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:11.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:11.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:11.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:12.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:12.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:12.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:12.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:12.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:12.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:13.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:13.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:13.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:13.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:14.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:14.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:14.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:14.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:15.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:15.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:15.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:15.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:16.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:16.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:16.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:16.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:17.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:17.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:17.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:17.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:17.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:17.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:18.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:18.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:18.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:18.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:19.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:19.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:19.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:19.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:20.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:20.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:20.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:20.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:21.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:21.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:21.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:21.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:22.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:22.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:22.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:22.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:22.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:22.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:23.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:23.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:23.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:23.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:24.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:24.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:24.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:24.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:25.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:25.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:25.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:25.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:26.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:26.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:26.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:26.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:27.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:27.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:27.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:27.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:27.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:27.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:28.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:28.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:28.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:28.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:29.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:29.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:29.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:29.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:30.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:30.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:30.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:30.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:31.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:31.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:31.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:31.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:32.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:32.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:32.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:32.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:32.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:32.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:33.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:33.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:33.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:33.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:34.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:34.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:34.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:34.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:35.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:35.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:35.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:35.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:36.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:36.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:36.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:36.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:37.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:37.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:37.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:37.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:37.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:37.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:38.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:38.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:39.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:39.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:39.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:39.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:40.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:40.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:40.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:40.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:41.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:41.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:41.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:41.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:42.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:42.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:42.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:42.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:42.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:42.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:43.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:43.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:43.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:43.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:44.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:44.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:44.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:44.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:45.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:45.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:45.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:45.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:46.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:46.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:46.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:46.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:47.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:47.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:47.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:47.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:47.743 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:47.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:48.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:48.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:48.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:48.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:49.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:49.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:49.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:49.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:50.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:50.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:50.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:50.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:51.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:51.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:51.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:51.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:52.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:52.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:52.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:52.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:52.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:52.739+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:53.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:53.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:53.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:53.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:54.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:54.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:54.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:54.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:55.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:55.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:55.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:56.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:56.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:56.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:56.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:57.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:57.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:57.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:57.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:57.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:12:57.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:12:58.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:58.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:58.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:58.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:12:59.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:59.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:12:59.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:12:59.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:00.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:00.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:00.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:00.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:01.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:01.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:01.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:01.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:02.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:02.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:02.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:02.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:02.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:02.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:03.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:03.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:03.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:03.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:04.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:04.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:04.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:04.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:05.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:05.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:05.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:05.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:06.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:06.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:06.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:06.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:07.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:07.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:07.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:07.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:07.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:07.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:08.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:08.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:08.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:08.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:09.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:09.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:09.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:09.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:10.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:10.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:10.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:10.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:11.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:11.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:11.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:11.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:12.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:12.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:12.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:12.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:12.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:12.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:13.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:13.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:13.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:13.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:14.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:14.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:14.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:14.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:15.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:15.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:15.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:15.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:16.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:16.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:16.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:16.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:17.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:17.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:17.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:17.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:17.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:17.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:18.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:18.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:18.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:18.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:19.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:19.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:19.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:19.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:20.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:20.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:20.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:20.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:21.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:21.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:21.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:21.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:22.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:22.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:22.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:22.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:22.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:23.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:23.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:23.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:23.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:24.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:24.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:24.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:24.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:25.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:25.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:25.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:25.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:26.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:26.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:26.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:26.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:27.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:27.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:27.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:27.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:27.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:27.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:28.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:28.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:28.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:28.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:29.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:29.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:29.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:29.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:30.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:30.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:30.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:30.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:31.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:31.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:31.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:31.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:32.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:32.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:32.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:32.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:33.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:33.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:33.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:33.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:34.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:34.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:34.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:34.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:35.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:35.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:35.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:35.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:36.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:36.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:36.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:36.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:37.750 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:37.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:37.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:37.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:37.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:37.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:38.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:38.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:38.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:38.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:39.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:39.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:39.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:40.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:40.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:40.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:40.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:41.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:41.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:41.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:41.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:42.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:42.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:42.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:42.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:42.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:42.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:43.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:43.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:43.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:43.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:44.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:44.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:44.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:44.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:45.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:45.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:45.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:45.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:46.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:46.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:46.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:46.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:47.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:47.745+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:47.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:47.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:47.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:47.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:48.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:48.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:48.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:48.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:49.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:49.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:49.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:49.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:50.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:50.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:50.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:50.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:51.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:51.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:51.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:51.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:52.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:52.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:52.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:52.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:52.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:52.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:53.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:53.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:53.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:53.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:54.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:54.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:54.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:54.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:55.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:55.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:55.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:55.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:56.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:56.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:56.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:56.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:57.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:13:57.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:13:57.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:57.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:57.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:57.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:59.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:59.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:59.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:59.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:13:59.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:59.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:13:59.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:13:59.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:01.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:01.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:01.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:01.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:01.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:01.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:01.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:01.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:02.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:02.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:03.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:03.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:03.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:03.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:03.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:03.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:03.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:03.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:05.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:05.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:05.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:05.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:05.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:05.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:05.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:05.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:06.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:06.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:06.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:06.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:07.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:07.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:08.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:08.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:08.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:08.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:08.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:08.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:08.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:08.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:09.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:09.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:09.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:09.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:10.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:10.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:10.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:10.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:11.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:11.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:11.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:11.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:12.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:12.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:12.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:12.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:12.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:12.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:13.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:13.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:13.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:13.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:14.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:14.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:14.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:14.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:15.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:15.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:16.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:15.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:16.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:16.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:16.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:17.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:17.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:17.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:17.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:17.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:17.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:18.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:18.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:18.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:18.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:19.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:19.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:19.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:19.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:20.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:20.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:20.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:20.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:21.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:21.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:21.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:21.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:22.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:22.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:22.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:22.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:22.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:22.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:23.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:23.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:23.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:23.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:24.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:24.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:24.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:24.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:25.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:25.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:26.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:26.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:26.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:26.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:27.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:27.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:27.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:27.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:27.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:27.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:28.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:28.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:28.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:28.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:29.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:29.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:29.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:29.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:30.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:30.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:30.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:30.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:31.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:31.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:31.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:31.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:32.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:32.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:32.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:32.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:32.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:32.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:33.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:33.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:33.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:33.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:34.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:34.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:34.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:34.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:35.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:35.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:35.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:35.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:36.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:36.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:37.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:36.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:37.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:37.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:37.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:37.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:37.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:37.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:38.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:38.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:38.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:38.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:39.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:39.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:39.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:39.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:40.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:40.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:40.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:40.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:42.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:42.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:42.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:42.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:42.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:42.756+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:43.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:43.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:43.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:43.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:44.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:44.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:44.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:44.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:44.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:44.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:44.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:44.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:46.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:46.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:46.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:46.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:46.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:46.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:46.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:46.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:47.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:47.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:47.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:47.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:47.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:47.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:48.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:48.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:48.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:48.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:49.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:49.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:49.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:49.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:50.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:50.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:50.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:50.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:51.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:51.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:51.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:51.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:52.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:52.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:52.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:52.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:52.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:52.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:53.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:53.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:53.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:53.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:54.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:54.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:54.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:54.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:55.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:55.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:55.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:55.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:56.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:56.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:56.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:56.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:57.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:14:57.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:14:57.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:57.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:57.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:57.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:58.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:58.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:58.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:58.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:14:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:59.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:14:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:14:59.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:00.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:00.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:00.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:00.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:01.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:01.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:01.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:01.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:02.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:02.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:02.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:02.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:02.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:02.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:03.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:03.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:03.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:03.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:04.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:04.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:04.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:04.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:05.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:05.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:05.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:05.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:06.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:06.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:06.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:06.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:07.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:07.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:07.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:07.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:07.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:07.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:08.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:08.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:08.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:08.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:09.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:09.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:09.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:09.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:10.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:10.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:10.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:10.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:11.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:11.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:11.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:11.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:12.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:12.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:12.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:12.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:12.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:12.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:13.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:13.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:13.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:13.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:14.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:14.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:14.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:14.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:15.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:15.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:15.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:15.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:16.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:16.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:16.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:16.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:17.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:17.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:17.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:17.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:17.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:17.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:18.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:18.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:18.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:18.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:19.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:19.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:19.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:20.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:20.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:20.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:20.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:21.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:21.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:21.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:21.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:22.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:22.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:22.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:22.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:22.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:22.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:23.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:23.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:23.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:23.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:24.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:24.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:24.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:24.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:25.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:25.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:25.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:25.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:26.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:26.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:26.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:26.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:27.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:27.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:27.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:27.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:27.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:27.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:28.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:28.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:28.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:28.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:29.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:29.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:29.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:29.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:30.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:30.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:30.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:30.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:31.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:31.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:31.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:31.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:32.767 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:32.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:32.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:32.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:32.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:32.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:33.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:33.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:33.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:33.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:34.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:34.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:34.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:34.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:35.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:35.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:35.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:35.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:36.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:36.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:36.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:36.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:37.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:37.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:37.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:37.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:37.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:37.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:38.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:38.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:38.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:38.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:39.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:39.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:39.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:39.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:40.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:40.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:40.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:40.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:41.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:41.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:41.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:41.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:42.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:42.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:42.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:42.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:42.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:42.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:43.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:43.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:43.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:43.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:44.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:44.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:44.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:44.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:45.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:45.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:45.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:45.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:46.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:46.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:46.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:46.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:47.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:47.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:47.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:47.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:47.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:47.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:48.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:48.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:48.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:48.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:50.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:50.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:50.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:50.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:51.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:51.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:51.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:51.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:52.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:52.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:52.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:52.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:52.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:52.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:53.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:53.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:53.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:53.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:54.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:54.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:54.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:54.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:55.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:55.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:55.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:55.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:56.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:56.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:56.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:56.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:56.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:57.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:57.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:57.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:15:57.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:15:57.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:57.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:57.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:57.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:58.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:58.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:58.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:58.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:15:59.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:59.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:15:59.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:15:59.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:00.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:00.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:00.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:00.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:01.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:01.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:01.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:01.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:02.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:02.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:02.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:02.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:02.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:02.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:03.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:03.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:03.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:03.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:04.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:04.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:04.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:04.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:05.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:05.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:05.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:05.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:06.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:06.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:06.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:06.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:07.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:07.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:07.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:07.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:07.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:07.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:08.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:08.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:08.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:08.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:09.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:09.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:09.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:09.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:10.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:10.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:10.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:10.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:11.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:11.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:11.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:11.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:12.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:12.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:12.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:12.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:12.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:12.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:13.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:13.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:13.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:13.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:14.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:14.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:14.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:14.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:15.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:15.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:15.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:15.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:16.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:16.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:16.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:16.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:17.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:17.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:17.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:17.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:17.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:17.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:18.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:18.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:18.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:18.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:19.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:19.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:19.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:19.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:20.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:20.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:20.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:20.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:21.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:21.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:21.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:22.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:22.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:22.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:22.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:22.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:22.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:23.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:23.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:23.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:23.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:24.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:24.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:24.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:24.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:25.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:25.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:25.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:25.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:26.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:26.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:26.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:26.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:27.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:27.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:27.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:27.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:27.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:27.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:28.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:28.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:28.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:28.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:29.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:29.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:29.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:29.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:30.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:30.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:30.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:30.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:31.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:31.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:31.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:31.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:32.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:32.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:32.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:32.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:32.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:32.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:33.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:33.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:33.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:33.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:34.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:34.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:34.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:34.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:35.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:35.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:35.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:35.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:36.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:36.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:36.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:36.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:37.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:37.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:37.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:37.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:37.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:37.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:38.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:38.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:38.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:38.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:39.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:39.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:39.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:39.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:40.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:40.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:40.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:40.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:41.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:41.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:41.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:41.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:42.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:42.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:42.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:42.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:42.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:42.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:43.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:43.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:43.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:43.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:44.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:44.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:44.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:44.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:45.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:45.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:45.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:45.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:46.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:46.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:46.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:46.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:47.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:47.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:47.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:47.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:47.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:47.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:48.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:48.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:48.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:48.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:49.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:49.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:49.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:49.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:50.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:50.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:50.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:50.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:52.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:52.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:52.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:52.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:52.779 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:52.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:53.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:53.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:53.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:53.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:54.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:54.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:54.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:54.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:55.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:55.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:56.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:56.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:56.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:57.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:57.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:57.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:57.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:57.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:16:57.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:16:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:58.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:58.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:16:59.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:59.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:16:59.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:16:59.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:00.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:00.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:00.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:00.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:01.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:01.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:01.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:01.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:02.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:02.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:02.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:02.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:02.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:02.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:03.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:03.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:03.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:03.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:04.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:04.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:04.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:04.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:05.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:05.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:05.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:05.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:06.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:06.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:06.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:06.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:07.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:07.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:07.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:07.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:07.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:07.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:08.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:08.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:08.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:08.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:09.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:09.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:09.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:09.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:10.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:10.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:10.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:10.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:11.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:11.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:11.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:11.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:12.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:12.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:12.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:12.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:12.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:12.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:13.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:13.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:13.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:13.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:14.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:14.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:14.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:14.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:15.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:15.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:15.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:15.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:16.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:16.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:16.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:16.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:17.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:17.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:17.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:17.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:17.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:17.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:18.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:18.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:18.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:18.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:19.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:19.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:19.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:19.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:20.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:20.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:20.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:20.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:21.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:21.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:21.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:21.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:22.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:22.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:22.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:22.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:22.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:22.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:23.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:23.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:23.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:23.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:24.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:24.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:24.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:24.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:25.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:25.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:25.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:25.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:26.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:26.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:26.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:26.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:27.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:27.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:27.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:27.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:27.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:27.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:28.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:28.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:28.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:28.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:29.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:29.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:29.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:29.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:30.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:30.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:30.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:30.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:31.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:31.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:31.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:31.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:32.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:32.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:32.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:32.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:32.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:32.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:33.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:33.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:33.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:33.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:34.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:34.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:34.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:34.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:35.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:35.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:35.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:35.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:36.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:36.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:36.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:36.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:37.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:37.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:37.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:37.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:37.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:37.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:38.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:38.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:38.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:38.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:39.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:39.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:39.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:39.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:40.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:40.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:40.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:40.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:41.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:41.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:41.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:42.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:42.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:42.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:42.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:42.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:42.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:43.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:43.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:43.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:43.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:44.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:44.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:44.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:44.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:45.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:45.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:45.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:45.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:46.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:46.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:46.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:46.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:47.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:47.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:47.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:47.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:47.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:47.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:48.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:48.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:48.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:48.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:49.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:49.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:49.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:49.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:50.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:50.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:50.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:50.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:51.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:51.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:51.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:51.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:52.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:52.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:52.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:52.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:52.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:52.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:53.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:53.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:53.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:53.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:54.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:54.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:54.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:54.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:55.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:55.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:55.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:55.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:56.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:56.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:56.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:56.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:57.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:57.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:57.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:57.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:57.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:17:57.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:17:58.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:58.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:58.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:58.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:17:59.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:59.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:17:59.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:17:59.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:00.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:00.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:00.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:00.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:01.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:01.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:01.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:01.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:02.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:02.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:02.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:02.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:02.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:02.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:03.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:03.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:03.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:03.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:04.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:04.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:04.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:04.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:05.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:05.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:05.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:05.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:06.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:06.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:06.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:06.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:07.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:07.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:07.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:07.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:07.790 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:07.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:08.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:08.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:08.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:08.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:09.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:09.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:09.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:09.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:10.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:10.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:10.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:10.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:11.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:11.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:11.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:11.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:12.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:12.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:12.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:12.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:12.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:12.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:13.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:13.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:13.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:13.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:14.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:14.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:14.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:14.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:15.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:15.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:15.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:15.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:16.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:16.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:16.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:16.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:17.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:17.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:17.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:17.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:17.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:17.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:18.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:18.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:18.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:18.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:19.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:19.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:19.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:19.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:20.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:20.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:20.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:20.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:21.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:21.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:21.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:21.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:22.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:22.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:22.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:22.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:22.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:22.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:23.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:23.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:23.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:23.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:24.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:24.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:24.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:24.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:25.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:25.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:25.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:25.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:26.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:26.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:26.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:26.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:27.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:27.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:27.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:27.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:27.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:27.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:28.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:28.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:28.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:28.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:29.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:29.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:29.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:29.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:30.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:30.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:30.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:30.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:31.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:31.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:31.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:31.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:32.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:32.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:32.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:32.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:32.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:32.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:33.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:33.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:33.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:33.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:34.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:34.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:34.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:34.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:35.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:35.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:35.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:35.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:36.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:36.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:36.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:37.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:37.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:37.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:37.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:37.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:37.789+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:38.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:38.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:38.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:38.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:39.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:39.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:39.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:39.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:40.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:40.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:40.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:40.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:41.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:41.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:41.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:41.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:42.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:42.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:42.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:42.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:42.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:42.789+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:43.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:43.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:43.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:43.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:44.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:44.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:44.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:44.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:45.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:45.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:45.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:45.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:46.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:46.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:47.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:47.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:47.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:47.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:47.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:47.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:48.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:48.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:48.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:48.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:49.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:49.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:49.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:49.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:50.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:50.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:50.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:50.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:51.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:51.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:51.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:51.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:52.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:52.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:52.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:52.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:52.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:52.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:53.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:53.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:53.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:53.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:54.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:54.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:54.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:54.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:55.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:55.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:55.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:55.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:56.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:56.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:56.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:56.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:57.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:57.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:57.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:57.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:57.797 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:18:57.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:18:58.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:58.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:58.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:58.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:18:59.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:59.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:18:59.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:18:59.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:00.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:00.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:00.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:00.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:01.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:01.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:01.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:01.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:02.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:02.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:02.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:02.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:02.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:03.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:03.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:03.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:03.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:04.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:04.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:04.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:04.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:05.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:05.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:05.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:05.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:06.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:06.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:06.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:06.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:07.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:07.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:07.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:07.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:07.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:07.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:08.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:08.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:08.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:08.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:09.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:09.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:09.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:09.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:10.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:10.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:10.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:10.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:11.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:11.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:11.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:11.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:12.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:12.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:12.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:12.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:12.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:12.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:13.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:13.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:13.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:13.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:14.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:14.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:14.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:14.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:15.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:14.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:15.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:14.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:15.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:15.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:15.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:15.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:17.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:17.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:17.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:17.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:17.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:17.796+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:18.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:18.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:18.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:18.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:19.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:19.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:19.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:19.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:20.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:20.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:20.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:20.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:21.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:21.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:21.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:21.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:22.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:22.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:22.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:22.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:22.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:22.796+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:23.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:23.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:23.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:23.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:24.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:24.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:24.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:24.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:25.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:25.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:25.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:25.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:26.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:26.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:26.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:26.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:27.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:27.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:27.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:27.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:27.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:27.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:28.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:28.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:28.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:28.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:29.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:29.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:29.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:29.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:30.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:30.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:30.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:30.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:31.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:31.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:31.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:31.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:32.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:32.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:32.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:32.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:32.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:32.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:33.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:33.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:33.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:33.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:34.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:34.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:34.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:34.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:35.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:35.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:35.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:35.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:36.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:36.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:36.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:36.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:37.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:37.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:37.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:37.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:37.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:37.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:38.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:38.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:38.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:38.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:39.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:39.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:39.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:39.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:40.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:40.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:40.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:40.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:41.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:41.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:41.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:41.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:42.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:42.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:42.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:42.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:42.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:42.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:43.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:43.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:43.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:43.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:44.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:44.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:44.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:44.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:45.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:45.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:45.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:45.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:46.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:46.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:46.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:46.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:47.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:47.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:47.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:47.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:47.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:47.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:48.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:48.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:48.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:48.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:49.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:49.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:49.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:49.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:50.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:50.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:50.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:50.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:51.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:51.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:51.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:51.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:52.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:52.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:52.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:52.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:52.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:52.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:53.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:53.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:53.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:53.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:54.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:54.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:54.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:54.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:55.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:55.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:55.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:55.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:56.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:56.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:56.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:57.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:57.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:57.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:57.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:57.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:19:57.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:19:58.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:58.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:58.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:58.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:19:59.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:59.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:19:59.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:19:59.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:00.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:00.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:00.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:00.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:01.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:01.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:01.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:01.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:02.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:02.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:02.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:02.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:02.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:02.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:03.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:03.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:03.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:03.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:04.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:04.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:04.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:04.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:05.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:05.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:05.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:05.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:06.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:06.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:06.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:06.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:07.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:07.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:07.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:07.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:07.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:07.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:08.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:08.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:08.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:08.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:09.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:09.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:09.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:09.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:10.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:10.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:10.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:10.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:11.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:11.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:11.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:11.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:12.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:12.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:12.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:12.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:12.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:12.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:13.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:13.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:13.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:13.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:14.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:14.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:14.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:14.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:15.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:15.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:15.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:15.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:16.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:16.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:16.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:16.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:17.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:17.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:17.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:17.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:17.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:17.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:18.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:18.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:18.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:18.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:19.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:19.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:19.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:19.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:20.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:20.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:20.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:20.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:21.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:21.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:21.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:21.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:22.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:22.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:22.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:22.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:22.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:22.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:23.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:23.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:23.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:23.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:24.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:24.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:24.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:24.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:25.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:25.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:25.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:25.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:26.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:26.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:26.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:26.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:27.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:27.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:27.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:27.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:27.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:27.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:28.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:28.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:28.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:28.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:29.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:29.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:29.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:29.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:30.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:30.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:30.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:30.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:31.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:31.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:31.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:31.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:32.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:32.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:32.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:32.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:32.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:32.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:33.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:33.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:33.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:33.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:34.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:34.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:34.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:34.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:35.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:35.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:35.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:35.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:36.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:36.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:36.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:36.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:37.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:37.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:37.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:37.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:37.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:37.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:38.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:38.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:38.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:38.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:39.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:39.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:39.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:39.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:40.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:40.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:40.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:40.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:41.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:41.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:41.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:41.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:42.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:42.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:42.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:42.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:42.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:42.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:43.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:43.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:43.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:43.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:44.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:44.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:44.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:44.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:45.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:45.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:45.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:45.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:46.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:46.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:46.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:46.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:47.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:47.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:47.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:47.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:47.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:47.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:48.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:48.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:48.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:48.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:49.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:49.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:49.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:49.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:50.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:50.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:50.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:50.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:51.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:51.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:51.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:51.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:52.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:52.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:52.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:52.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:52.814 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:52.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:53.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:53.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:53.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:53.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:54.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:54.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:54.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:54.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:55.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:55.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:55.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:55.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:56.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:56.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:56.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:56.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:57.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:57.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:57.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:57.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:20:57.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:20:58.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:58.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:58.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:58.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:20:59.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:59.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:20:59.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:20:59.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:00.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:00.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:00.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:00.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:01.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:01.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:01.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:01.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:02.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:02.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:02.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:02.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:02.816 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:02.813+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:03.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:03.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:03.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:03.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:04.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:04.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:04.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:04.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:05.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:05.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:05.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:05.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:06.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:06.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:06.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:06.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:07.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:07.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:07.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:07.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:07.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:07.813+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:08.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:08.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:08.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:08.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:09.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:09.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:09.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:09.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:10.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:10.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:10.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:10.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:11.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:11.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:11.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:11.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:12.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:12.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:12.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:12.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:12.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:12.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:13.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:13.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:13.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:13.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:14.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:14.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:14.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:14.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:15.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:15.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:15.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:15.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:16.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:16.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:16.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:16.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:17.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:17.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:17.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:17.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:17.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:17.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:18.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:18.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:18.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:18.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:19.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:19.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:19.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:19.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:20.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:20.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:20.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:20.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:21.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:21.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:21.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:21.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:22.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:22.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:22.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:22.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:22.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:22.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:23.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:23.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:23.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:23.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:24.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:24.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:24.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:24.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:25.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:25.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:25.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:25.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:26.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:26.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:26.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:26.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:27.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:27.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:27.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:27.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:27.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:27.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:28.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:28.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:28.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:28.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:29.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:29.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:29.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:29.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:30.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:30.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:30.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:30.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:31.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:31.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:31.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:31.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:32.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:32.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:32.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:32.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:32.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:32.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:33.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:33.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:33.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:33.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:34.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:34.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:34.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:35.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:35.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:35.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:35.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:36.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:36.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:36.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:36.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:37.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:37.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:37.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:37.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:37.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:37.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:38.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:38.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:38.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:38.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:39.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:39.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:39.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:39.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:40.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:40.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:40.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:40.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:41.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:41.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:41.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:41.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:42.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:42.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:42.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:42.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:42.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:42.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:43.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:43.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:43.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:43.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:44.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:44.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:44.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:44.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:45.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:45.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:45.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:45.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:46.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:46.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:46.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:46.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:47.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:47.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:47.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:47.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:47.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:47.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:48.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:48.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:48.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:48.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:49.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:49.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:49.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:49.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:50.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:50.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:50.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:50.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:51.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:51.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:51.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:51.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:52.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:52.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:52.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:52.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:52.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:52.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:53.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:53.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:53.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:53.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:54.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:54.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:54.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:54.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:55.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:55.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:55.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:55.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:56.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:56.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:56.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:56.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:57.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:57.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:57.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:57.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:57.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:21:57.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:21:58.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:58.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:58.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:58.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:21:59.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:59.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:21:59.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:21:59.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:00.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:00.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:22:00.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:00.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:01.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:11:29.826145+0000 (oldest deadline 2019-10-17T21:11:49.826145+0000) 2019-10-17T21:22:01.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:01.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:02.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:02.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:02.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:02.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:03.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:03.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:04.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:04.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:05.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:05.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:06.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:06.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:07.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:07.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:07.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:07.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:08.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:08.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:09.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:09.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:10.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:10.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:11.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:11.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:12.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:12.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:12.826 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:12.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:13.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:13.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:14.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:14.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:15.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:15.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:16.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:16.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:17.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:17.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:17.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:17.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:18.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:18.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:19.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:19.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:20.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:20.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:21.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:21.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:22.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:22.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:22.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:22.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:23.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:23.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:24.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:24.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:25.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:25.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:26.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:26.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:27.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:27.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:27.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:27.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:27.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:27.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:28.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:28.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:28.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:28.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:29.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:29.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:29.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:29.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:30.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:30.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:30.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:30.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:31.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:31.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:32.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:32.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:32.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:32.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:32.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:32.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:33.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:33.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:33.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:33.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:34.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:34.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:34.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:34.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:35.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:35.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:35.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:35.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:36.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:36.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:36.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:36.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:37.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:37.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:37.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:37.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:37.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:37.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:38.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:38.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:38.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:38.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:39.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:39.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:39.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:39.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:40.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:40.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:40.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:40.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:41.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:41.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:41.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:41.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:42.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:42.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:42.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:42.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:42.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:42.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:43.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:43.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:43.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:43.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:44.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:44.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:44.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:44.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:45.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:45.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:45.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:45.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:46.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:46.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:46.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:46.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:47.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:47.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:47.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:47.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:47.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:47.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:48.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:48.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:48.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:48.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:49.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:49.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:49.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:50.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:50.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:50.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:50.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:51.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:51.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:51.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:51.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:52.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:52.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:52.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:52.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:52.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:52.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:53.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:53.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:53.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:53.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:54.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:54.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:54.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:55.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:55.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:55.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:55.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:56.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:56.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:56.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:56.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:57.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:57.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:57.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:57.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:57.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:22:57.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:22:58.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:58.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:58.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:58.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:22:59.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:59.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:22:59.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:22:59.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:00.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:00.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:00.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:00.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:01.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:01.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:01.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:01.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:02.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:02.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:02.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:02.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:02.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:02.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:03.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:03.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:03.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:03.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:04.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:04.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:04.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:04.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:05.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:05.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:05.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:05.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:06.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:06.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:06.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:06.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:07.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:07.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:07.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:07.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:07.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:07.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:08.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:08.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:08.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:08.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:09.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:09.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:09.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:09.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:10.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:10.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:10.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:10.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:11.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:11.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:11.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:11.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:12.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:12.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:12.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:12.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:12.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:12.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:13.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:13.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:13.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:13.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:14.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:14.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:14.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:14.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:15.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:15.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:15.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:15.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:16.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:16.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:16.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:16.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:17.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:17.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:17.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:17.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:17.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:17.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:18.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:18.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:18.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:18.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:19.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:19.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:19.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:19.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:20.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:20.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:20.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:20.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:21.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:21.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:21.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:21.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:22.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:22.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:22.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:22.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:22.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:22.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:23.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:23.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:23.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:23.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:24.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:24.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:24.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:24.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:25.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:25.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:25.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:25.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:26.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:26.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:26.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:26.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:27.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:27.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:27.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:27.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:27.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:27.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:28.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:28.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:28.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:28.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:29.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:29.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:29.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:29.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:30.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:30.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:30.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:30.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:31.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:31.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:31.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:31.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:32.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:32.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:32.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:32.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:32.839 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:32.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:33.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:33.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:33.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:33.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:34.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:34.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:34.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:34.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:35.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:35.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:35.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:35.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:36.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:36.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:36.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:36.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:37.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:37.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:37.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:37.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:37.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:37.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:38.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:38.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:38.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:38.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:39.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:39.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:39.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:39.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:40.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:40.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:40.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:40.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:41.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:41.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:41.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:41.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:42.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:42.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:42.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:42.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:42.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:42.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:43.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:43.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:43.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:43.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:44.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:44.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:44.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:44.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:45.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:45.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:45.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:45.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:46.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:46.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:46.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:46.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:47.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:47.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:47.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:47.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:47.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:47.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:48.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:48.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:48.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:48.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:49.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:49.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:49.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:49.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:50.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:50.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:50.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:50.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:51.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:51.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:51.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:51.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:52.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:52.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:52.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:52.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:52.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:52.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:53.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:53.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:53.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:53.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:54.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:54.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:54.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:54.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:55.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:55.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:55.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:55.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:56.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:56.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:56.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:56.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:57.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:57.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:57.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:57.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:57.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:23:57.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:23:58.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:58.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:58.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:58.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:23:59.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:59.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:23:59.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:23:59.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:00.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:00.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:00.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:00.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:01.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:01.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:01.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:01.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:02.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:02.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:02.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:02.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:02.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:02.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:03.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:03.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:03.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:03.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:04.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:04.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:04.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:04.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:05.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:05.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:05.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:05.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:06.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:06.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:06.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:06.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:07.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:07.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:07.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:07.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:07.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:07.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:08.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:08.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:08.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:08.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:09.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:09.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:09.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:09.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:10.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:10.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:10.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:10.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:11.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:11.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:11.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:11.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:12.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:12.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:12.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:12.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:12.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:12.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:13.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:13.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:13.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:13.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:14.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:14.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:14.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:14.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:15.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:15.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:16.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:16.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:16.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:16.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:17.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:17.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:17.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:17.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:17.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:17.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:19.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:18.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:19.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:18.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:20.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:20.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:20.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:20.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:21.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:21.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:21.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:21.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:22.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:22.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:22.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:22.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:22.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:22.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:23.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:23.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:23.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:23.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:24.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:24.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:24.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:24.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:25.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:25.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:25.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:25.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:26.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:26.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:26.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:26.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:27.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:27.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:27.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:27.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:27.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:27.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:28.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:28.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:28.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:28.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:29.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:29.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:29.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:29.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:30.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:30.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:30.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:30.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:31.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:31.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:31.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:31.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:32.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:32.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:32.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:32.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:32.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:32.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:33.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:33.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:33.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:33.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:34.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:34.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:34.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:34.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:35.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:35.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:35.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:35.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:36.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:36.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:36.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:36.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:37.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:37.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:37.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:37.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:37.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:37.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:38.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:38.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:38.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:38.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:39.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:39.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:39.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:39.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:40.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:40.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:40.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:40.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:41.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:41.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:41.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:41.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:42.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:42.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:42.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:42.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:42.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:42.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:43.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:43.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:43.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:43.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:44.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:44.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:44.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:44.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:45.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:45.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:45.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:45.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:46.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:46.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:46.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:46.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:47.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:47.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:47.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:47.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:47.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:47.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:48.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:48.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:48.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:48.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:49.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:49.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:49.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:49.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:50.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:50.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:50.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:50.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:51.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:51.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:51.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:51.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:52.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:52.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:52.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:52.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:52.853 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:52.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:53.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:53.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:53.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:53.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:54.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:54.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:54.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:54.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:55.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:55.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:55.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:55.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:56.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:56.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:56.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:56.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:57.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:57.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:57.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:57.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:57.853 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:24:57.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:24:58.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:58.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:58.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:58.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:24:59.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:59.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:24:59.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:24:59.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:00.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:00.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:00.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:00.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:01.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:01.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:01.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:01.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:02.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:02.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:02.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:02.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:02.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:02.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:03.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:03.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:03.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:03.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:04.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:04.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:04.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:04.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:05.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:05.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:05.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:05.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:06.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:06.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:06.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:06.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:06.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:06.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:06.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:06.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:07.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:07.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:08.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:07.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:08.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:07.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:09.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:09.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:09.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:09.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:09.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:09.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:09.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:09.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:10.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:10.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:10.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:10.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:11.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:11.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:11.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:11.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:12.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:12.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:12.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:12.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:12.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:12.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:13.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:13.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:13.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:13.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:14.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:14.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:14.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:14.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:15.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:15.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:15.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:15.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:16.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:16.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:16.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:16.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:17.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:17.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:17.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:17.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:17.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:17.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:18.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:18.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:18.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:18.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:19.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:19.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:19.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:19.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:20.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:20.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:20.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:20.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:21.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:21.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:21.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:22.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:22.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:22.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:22.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:22.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:22.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:23.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:23.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:23.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:23.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:24.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:24.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:24.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:24.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:25.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:25.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:25.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:25.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:26.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:26.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:26.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:26.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:27.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:27.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:27.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:27.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:27.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:27.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:28.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:28.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:28.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:28.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:29.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:29.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:29.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:29.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:30.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:30.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:30.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:30.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:31.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:31.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:31.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:31.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:32.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:32.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:32.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:32.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:32.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:32.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:33.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:33.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:33.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:33.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:34.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:34.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:34.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:34.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:35.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:35.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:35.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:35.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:36.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:36.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:36.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:36.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:37.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:37.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:37.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:37.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:37.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:37.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:38.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:38.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:38.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:38.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:39.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:39.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:39.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:39.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:40.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:40.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:40.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:40.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:41.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:41.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:41.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:41.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:42.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:42.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:42.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:42.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:42.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:42.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:43.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:43.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:43.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:43.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:44.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:44.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:44.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:44.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:45.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:45.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:45.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:45.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:46.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:46.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:46.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:46.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:47.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:47.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:47.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:47.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:47.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:47.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:48.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:48.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:48.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:48.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:49.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:49.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:49.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:49.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:50.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:50.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:50.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:50.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:51.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:51.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:51.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:51.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:52.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:52.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:52.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:52.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:52.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:52.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:53.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:53.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:53.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:53.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:54.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:54.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:54.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:54.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:55.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:55.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:55.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:55.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:56.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:56.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:56.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:56.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:57.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:57.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:57.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:57.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:57.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:25:57.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:25:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:58.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:58.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:58.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:25:59.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:59.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:25:59.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:25:59.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:00.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:00.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:00.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:00.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:01.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:01.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:01.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:01.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:02.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:02.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:02.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:02.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:02.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:02.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:03.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:03.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:03.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:03.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:04.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:04.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:04.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:04.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:05.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:05.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:06.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:06.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:06.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:06.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:07.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:07.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:07.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:07.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:07.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:07.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:08.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:08.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:08.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:08.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:09.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:09.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:09.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:09.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:10.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:10.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:10.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:10.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:11.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:11.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:11.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:11.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:12.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:12.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:12.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:12.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:12.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:12.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:13.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:13.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:13.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:13.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:14.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:14.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:14.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:14.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:15.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:15.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:15.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:15.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:16.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:16.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:16.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:16.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:17.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:17.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:17.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:17.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:17.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:17.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:18.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:18.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:18.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:18.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:19.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:19.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:19.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:19.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:20.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:20.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:20.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:20.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:21.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:21.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:21.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:21.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:22.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:22.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:22.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:22.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:22.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:22.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:23.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:23.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:23.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:23.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:24.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:24.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:24.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:24.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:25.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:25.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:25.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:25.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:26.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:26.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:26.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:26.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:27.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:27.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:27.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:27.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:27.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:27.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:28.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:28.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:28.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:28.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:29.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:29.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:29.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:29.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:30.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:30.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:30.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:30.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:31.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:31.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:31.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:31.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:32.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:32.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:32.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:32.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:32.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:32.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:33.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:33.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:33.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:33.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:34.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:34.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:34.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:34.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:35.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:35.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:35.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:35.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:36.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:36.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:36.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:36.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:37.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:37.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:37.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:37.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:37.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:37.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:38.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:38.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:38.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:38.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:39.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:39.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:39.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:39.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:40.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:40.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:40.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:40.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:41.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:41.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:41.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:41.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:42.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:42.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:42.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:42.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:42.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:42.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:43.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:43.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:43.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:43.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:44.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:44.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:44.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:44.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:45.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:45.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:45.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:45.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:46.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:46.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:46.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:46.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:47.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:47.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:47.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:47.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:47.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:47.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:48.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:48.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:48.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:48.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:49.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:49.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:49.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:49.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:50.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:50.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:50.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:50.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:51.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:51.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:51.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:51.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:52.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:52.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:52.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:52.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:52.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:52.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:53.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:53.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:53.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:53.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:54.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:54.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:54.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:54.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:55.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:55.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:55.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:55.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:56.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:56.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:56.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:56.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:57.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:57.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:57.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:26:57.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:26:58.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:58.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:58.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:58.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:26:59.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:59.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:26:59.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:26:59.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:00.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:00.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:00.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:00.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:01.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:01.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:01.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:01.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:02.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:02.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:02.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:02.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:02.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:03.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:03.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:03.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:03.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:04.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:04.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:04.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:04.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:05.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:05.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:05.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:05.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:06.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:06.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:06.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:06.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:07.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:07.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:07.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:07.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:07.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:07.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:08.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:08.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:08.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:08.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:09.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:09.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:09.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:09.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:10.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:10.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:10.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:10.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:11.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:11.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:11.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:11.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:12.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:12.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:12.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:12.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:12.875 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:12.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:13.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:13.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:13.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:13.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:14.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:14.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:14.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:14.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:15.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:15.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:15.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:15.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:16.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:16.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:16.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:16.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:17.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:17.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:17.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:17.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:17.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:17.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:18.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:18.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:18.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:18.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:19.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:19.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:19.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:19.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:20.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:20.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:20.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:20.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:21.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:21.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:21.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:21.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:22.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:22.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:22.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:22.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:22.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:22.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:23.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:23.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:23.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:23.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:24.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:24.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:24.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:24.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:24.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:24.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:24.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:24.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:26.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:25.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:26.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:25.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:27.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:27.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:27.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:27.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:27.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:27.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:28.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:28.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:28.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:28.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:29.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:28.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:29.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:28.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:29.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:29.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:29.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:29.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:31.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:31.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:31.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:31.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:32.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:32.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:32.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:32.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:32.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:32.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:33.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:33.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:34.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:34.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:34.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:34.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:35.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:35.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:35.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:35.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:36.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:36.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:36.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:36.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:37.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:37.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:37.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:37.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:37.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:37.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:38.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:38.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:38.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:38.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:39.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:39.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:39.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:39.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:40.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:40.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:40.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:40.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:41.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:41.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:41.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:41.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:42.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:42.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:42.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:42.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:42.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:42.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:43.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:43.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:43.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:43.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:44.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:44.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:44.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:44.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:45.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:45.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:45.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:45.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:46.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:46.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:46.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:46.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:47.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:47.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:47.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:47.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:47.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:47.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:47.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:47.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:47.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:47.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:48.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:48.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:48.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:48.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:49.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:49.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:49.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:49.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:50.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:50.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:50.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:50.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:51.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:51.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:51.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:51.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:52.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:52.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:52.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:52.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:52.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:52.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:53.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:53.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:53.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:53.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:54.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:54.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:54.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:54.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:55.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:55.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:55.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:55.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:56.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:56.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:56.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:56.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:57.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:27:57.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:27:58.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:58.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:58.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:58.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:27:59.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:59.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:27:59.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:27:59.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:00.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:00.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:00.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:00.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:01.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:01.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:01.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:01.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:02.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:02.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:02.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:02.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:02.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:02.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:02.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:02.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:02.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:02.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:04.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:04.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:04.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:04.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:04.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:04.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:04.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:04.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:05.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:05.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:05.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:05.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:07.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:06.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:07.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:06.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:07.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:07.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:08.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:08.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:08.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:08.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:08.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:08.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:08.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:08.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:10.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:10.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:10.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:10.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:11.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:11.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:11.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:11.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:12.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:12.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:12.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:12.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:13.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:13.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:13.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:13.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:14.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:14.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:14.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:14.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:15.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:15.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:15.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:15.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:16.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:16.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:16.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:16.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:17.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:17.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:17.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:17.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:17.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:17.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:18.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:18.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:18.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:18.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:19.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:19.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:19.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:19.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:19.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:19.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:19.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:19.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:20.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:20.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:20.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:20.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:21.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:21.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:21.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:21.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:22.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:22.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:22.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:22.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:22.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:22.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:23.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:23.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:23.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:23.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:25.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:25.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:25.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:25.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:26.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:26.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:26.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:26.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:27.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:27.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:27.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:27.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:27.887 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:27.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:27.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:27.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:27.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:27.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:28.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:28.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:28.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:28.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:29.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:29.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:29.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:29.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:30.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:30.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:30.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:30.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:31.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:31.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:31.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:31.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:32.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:32.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:32.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:32.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:32.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:32.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:33.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:33.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:33.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:33.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:34.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:34.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:34.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:34.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:35.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:35.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:35.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:35.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:36.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:36.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:36.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:36.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:37.889 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:37.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:37.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:37.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:37.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:37.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:38.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:38.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:38.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:38.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:39.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:39.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:39.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:39.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:40.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:40.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:40.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:40.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:41.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:41.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:41.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:41.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:42.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:42.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:42.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:42.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:42.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:42.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:43.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:43.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:43.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:43.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:44.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:44.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:44.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:44.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:46.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:46.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:46.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:46.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:47.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:47.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:47.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:47.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:47.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:47.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:48.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:48.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:48.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:48.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:49.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:49.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:49.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:49.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:50.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:50.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:50.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:50.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:51.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:51.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:51.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:51.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:52.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:52.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:52.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:52.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:52.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:52.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:53.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:52.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:53.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:52.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:54.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:54.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:54.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:54.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:55.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:54.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:55.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:54.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:55.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:55.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:55.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:55.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:56.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:56.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:56.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:56.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:57.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:28:57.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:28:57.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:57.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:57.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:57.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:28:58.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:58.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:28:58.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:28:58.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:00.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:00.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:00.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:00.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:01.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:01.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:01.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:01.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:02.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:02.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:02.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:02.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:02.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:02.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:03.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:03.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:03.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:03.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:04.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:04.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:04.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:04.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:05.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:05.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:06.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:05.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:06.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:05.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:07.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:07.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:07.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:07.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:07.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:07.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:08.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:08.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:08.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:08.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:09.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:09.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:09.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:09.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:10.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:10.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:10.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:10.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:11.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:11.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:11.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:11.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:12.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:12.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:12.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:12.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:12.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:12.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:13.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:13.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:13.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:13.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:14.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:14.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:14.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:14.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:15.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:15.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:15.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:15.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:16.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:16.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:16.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:16.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:17.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:17.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:17.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:17.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:17.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:17.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:18.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:18.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:18.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:18.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:19.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:19.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:19.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:19.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:20.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:20.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:20.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:20.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:21.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:21.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:21.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:21.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:22.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:22.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:22.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:22.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:22.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:22.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:23.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:23.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:23.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:23.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:24.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:24.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:24.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:24.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:25.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:25.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:25.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:25.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:26.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:26.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:26.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:26.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:27.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:27.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:27.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:27.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:27.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:27.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:28.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:28.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:28.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:28.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:29.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:29.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:29.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:29.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:30.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:30.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:30.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:30.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:31.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:31.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:31.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:31.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:32.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:32.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:32.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:32.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:32.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:32.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:33.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:33.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:33.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:33.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:34.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:34.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:34.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:34.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:35.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:35.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:35.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:35.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:36.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:36.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:36.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:36.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:37.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:37.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:37.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:37.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:37.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:37.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:38.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:38.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:38.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:38.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:39.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:39.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:39.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:39.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:40.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:40.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:40.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:40.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:41.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:41.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:41.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:41.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:42.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:42.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:42.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:42.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:42.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:42.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:43.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:43.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:43.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:43.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:44.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:44.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:44.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:44.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:45.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:45.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:45.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:45.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:46.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:46.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:46.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:46.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:47.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:47.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:47.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:47.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:47.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:47.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:48.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:48.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:48.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:48.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:49.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:49.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:49.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:49.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:50.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:50.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:50.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:50.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:51.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:51.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:51.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:51.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:52.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:52.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:52.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:52.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:52.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:52.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:53.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:53.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:53.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:53.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:54.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:54.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:54.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:54.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:55.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:55.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:55.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:55.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:56.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:56.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:56.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:56.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:57.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:57.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:57.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:57.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:57.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:29:57.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:29:58.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:58.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:58.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:58.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:29:59.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:59.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:29:59.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:29:59.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:00.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:00.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:00.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:00.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:01.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:01.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:01.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:01.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:02.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:02.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:02.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:02.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:02.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:02.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:03.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:03.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:03.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:03.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:04.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:04.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:04.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:04.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:05.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:05.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:06.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:06.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:06.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:06.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:07.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:07.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:07.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:07.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:07.904 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:07.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:08.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:08.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:08.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:08.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:09.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:09.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:09.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:09.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:10.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:10.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:10.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:10.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:11.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:11.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:11.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:11.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:11.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:11.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:11.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:11.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:12.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:12.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:13.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:13.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:13.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:13.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:13.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:13.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:13.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:13.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:14.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:14.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:14.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:14.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:15.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:15.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:15.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:15.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:16.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:16.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:16.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:16.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:17.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:17.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:17.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:17.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:17.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:17.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:18.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:18.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:18.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:19.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:19.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:19.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:19.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:20.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:20.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:20.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:20.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:21.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:21.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:21.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:21.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:22.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:22.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:22.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:22.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:22.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:22.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:23.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:23.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:23.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:23.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:24.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:24.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:24.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:24.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:25.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:25.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:25.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:25.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:26.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:26.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:26.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:26.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:27.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:27.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:27.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:27.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:27.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:27.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:28.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:28.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:28.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:28.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:29.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:29.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:29.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:29.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:30.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:30.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:30.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:30.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:31.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:31.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:31.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:31.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:32.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:32.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:32.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:32.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:32.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:32.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:33.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:33.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:33.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:33.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:34.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:34.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:34.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:34.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:35.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:35.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:35.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:35.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:36.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:36.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:36.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:36.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:37.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:37.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:37.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:37.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:37.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:37.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:38.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:38.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:38.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:38.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:39.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:39.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:39.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:39.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:40.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:40.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:40.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:40.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:41.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:41.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:41.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:41.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:42.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:42.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:42.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:42.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:42.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:42.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:43.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:43.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:43.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:43.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:44.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:44.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:44.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:44.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:45.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:45.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:45.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:45.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:46.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:46.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:46.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:46.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:47.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:47.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:47.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:47.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:47.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:47.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:48.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:48.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:48.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:48.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:49.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:49.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:49.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:49.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:50.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:50.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:50.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:50.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:51.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:51.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:51.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:51.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:52.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:52.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:52.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:52.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:52.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:52.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:53.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:53.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:53.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:53.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:54.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:54.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:54.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:54.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:55.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:55.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:55.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:55.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:56.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:56.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:56.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:56.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:57.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:57.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:57.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:57.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:57.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:30:57.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:30:58.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:58.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:58.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:58.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:30:59.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:59.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:30:59.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:30:59.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:00.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:00.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:00.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:00.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:01.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:01.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:01.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:01.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:02.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:02.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:02.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:02.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:02.913 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:02.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:03.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:03.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:03.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:03.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:04.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:04.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:05.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:05.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:05.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:05.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:06.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:06.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:06.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:06.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:07.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:07.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:07.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:07.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:07.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:07.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:08.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:08.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:08.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:08.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:09.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:09.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:09.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:09.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:10.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:10.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:10.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:10.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:11.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:11.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:11.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:11.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:12.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:12.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:12.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:12.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:12.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:12.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:13.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:13.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:13.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:13.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:14.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:14.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:14.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:14.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:15.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:15.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:15.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:15.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:16.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:16.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:16.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:16.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:17.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:17.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:17.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:17.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:17.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:18.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:18.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:18.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:18.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:19.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:19.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:19.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:19.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:20.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:20.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:20.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:20.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:21.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:21.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:21.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:21.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:22.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:22.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:22.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:22.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:22.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:22.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:23.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:23.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:23.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:23.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:24.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:24.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:24.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:24.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:25.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:25.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:25.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:25.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:26.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:26.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:26.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:26.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:27.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:27.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:27.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:27.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:27.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:27.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:28.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:28.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:28.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:28.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:29.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:29.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:29.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:29.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:30.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:30.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:30.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:30.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:31.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:31.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:32.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:32.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:32.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:32.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:32.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:32.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:33.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:33.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:33.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:33.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:34.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:34.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:34.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:34.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:35.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:35.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:35.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:35.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:36.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:36.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:36.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:36.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:37.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:37.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:37.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:37.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:37.919 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:37.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:38.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:38.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:38.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:38.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:39.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:39.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:39.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:39.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:40.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:40.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:40.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:40.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:41.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:41.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:41.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:41.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:42.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:42.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:42.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:42.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:42.919 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:42.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:43.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:43.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:43.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:43.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:44.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:44.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:44.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:44.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:45.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:45.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:45.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:45.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:46.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:46.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:46.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:46.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:47.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:47.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:47.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:47.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:47.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:47.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:48.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:48.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:48.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:48.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:49.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:49.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:49.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:49.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:50.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:50.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:50.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:50.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:51.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:51.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:51.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:51.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:52.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:52.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:52.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:52.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:52.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:52.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:53.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:53.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:53.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:53.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:54.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:54.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:54.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:54.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:55.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:55.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:55.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:55.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:56.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:56.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:56.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:56.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:57.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:57.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:57.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:57.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:57.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:31:57.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:31:58.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:58.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:58.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:58.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:31:59.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:59.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:31:59.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:31:59.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:00.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:00.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:00.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:00.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:01.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:01.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:01.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:01.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:02.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:02.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:02.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:02.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:02.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:02.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:03.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:03.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:03.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:03.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:04.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:04.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:04.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:04.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:05.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:05.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:05.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:05.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:06.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:06.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:06.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:06.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:07.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:07.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:07.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:07.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:07.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:07.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:08.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:08.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:08.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:08.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:09.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:09.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:09.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:09.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:10.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:10.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:10.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:10.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:11.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:11.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:11.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:11.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:12.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:12.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:12.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:12.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:12.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:12.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:13.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:13.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:13.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:13.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:14.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:14.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:14.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:14.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:15.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:15.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:15.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:15.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:16.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:16.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:16.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:16.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:17.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:17.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:17.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:17.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:17.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:17.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:18.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:18.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:18.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:18.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:19.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:19.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:19.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:19.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:20.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:20.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:20.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:20.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:21.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:21.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:21.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:21.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:22.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:22.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:22.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:22.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:22.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:22.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:23.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:23.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:23.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:23.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:24.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:24.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:24.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:24.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:25.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:25.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:25.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:25.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:26.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:26.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:26.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:26.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:27.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:27.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:27.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:27.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:28.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:28.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:28.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:28.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:29.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:29.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:29.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:29.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:30.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:30.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:30.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:30.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:31.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:31.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:31.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:31.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:32.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:32.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:32.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:32.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:32.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:32.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:33.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:33.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:33.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:33.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:34.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:34.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:34.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:34.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:35.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:35.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:35.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:35.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:36.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:36.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:36.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:36.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:37.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:37.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:37.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:37.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:37.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:37.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:38.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:38.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:38.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:38.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:39.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:39.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:22:07.052945+0000 (oldest deadline 2019-10-17T21:22:27.052945+0000) 2019-10-17T21:32:39.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:39.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:40.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:40.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:41.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:41.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:42.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:42.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:42.929 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:42.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:43.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:43.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:44.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:44.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:45.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:45.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:46.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:46.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:47.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:47.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:47.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:47.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:48.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:48.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:49.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:49.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:50.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:50.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:51.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:51.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:52.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:52.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:52.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:52.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:53.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:53.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:54.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:54.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:55.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:55.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:56.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:56.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:57.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:57.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:57.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:32:57.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:32:58.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:58.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:32:59.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:32:59.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:00.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:00.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:01.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:01.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:01.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:01.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:02.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:02.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:02.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:02.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:02.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:02.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:03.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:03.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:03.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:03.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:04.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:04.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:04.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:04.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:05.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:05.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:05.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:05.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:06.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:06.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:06.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:06.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:07.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:07.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:07.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:07.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:07.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:07.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:08.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:08.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:08.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:08.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:09.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:09.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:09.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:09.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:10.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:10.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:10.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:10.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:11.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:11.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:11.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:11.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:12.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:12.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:12.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:12.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:12.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:12.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:13.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:13.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:13.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:13.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:14.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:14.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:14.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:14.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:15.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:15.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:15.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:15.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:16.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:16.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:16.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:16.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:17.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:17.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:17.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:17.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:17.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:17.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:18.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:18.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:18.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:18.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:19.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:19.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:19.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:19.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:20.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:20.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:20.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:20.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:21.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:21.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:21.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:21.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:22.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:22.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:22.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:22.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:22.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:22.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:23.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:23.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:23.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:23.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:24.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:24.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:24.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:24.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:25.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:25.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:25.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:25.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:26.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:26.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:26.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:26.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:27.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:27.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:27.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:27.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:27.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:27.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:28.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:28.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:28.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:28.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:29.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:29.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:29.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:29.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:30.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:30.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:30.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:30.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:31.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:31.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:31.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:31.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:32.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:32.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:32.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:32.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:32.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:32.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:33.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:33.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:33.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:33.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:34.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:34.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:34.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:34.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:35.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:35.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:35.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:35.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:36.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:36.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:36.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:36.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:37.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:37.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:37.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:37.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:37.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:37.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:38.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:38.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:38.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:38.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:39.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:39.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:39.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:39.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:40.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:40.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:40.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:40.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:41.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:41.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:41.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:41.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:42.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:42.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:42.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:42.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:42.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:42.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:43.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:43.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:43.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:43.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:44.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:44.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:44.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:44.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:45.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:45.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:45.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:45.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:46.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:46.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:46.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:46.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:47.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:47.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:47.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:47.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:47.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:47.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:48.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:48.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:48.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:48.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:49.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:49.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:49.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:49.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:50.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:50.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:50.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:50.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:51.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:51.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:51.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:51.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:52.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:52.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:52.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:52.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:52.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:52.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:53.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:53.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:53.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:53.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:54.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:54.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:54.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:54.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:55.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:55.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:55.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:55.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:56.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:56.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:56.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:56.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:57.942 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:33:57.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:33:58.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:57.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:58.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:57.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:33:59.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:59.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:33:59.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:33:59.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:00.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:00.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:00.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:00.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:01.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:01.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:01.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:01.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:02.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:02.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:02.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:02.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:02.942 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:02.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:03.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:03.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:03.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:03.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:04.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:04.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:04.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:04.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:05.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:05.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:05.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:05.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:06.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:06.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:06.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:06.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:07.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:07.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:07.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:07.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:07.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:07.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:08.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:08.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:08.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:08.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:09.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:09.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:09.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:09.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:10.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:10.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:10.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:10.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:11.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:11.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:11.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:11.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:11.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:11.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:11.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:11.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:12.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:12.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:13.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:13.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:13.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:13.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:14.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:14.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:14.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:14.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:15.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:15.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:15.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:15.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:16.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:16.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:16.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:16.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:17.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:17.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:17.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:17.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:17.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:17.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:18.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:18.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:18.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:18.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:19.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:19.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:19.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:19.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:20.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:20.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:20.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:20.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:21.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:21.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:21.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:21.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:22.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:22.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:22.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:22.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:22.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:22.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:23.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:23.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:23.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:23.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:23.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:23.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:25.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:24.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:25.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:24.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:26.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:26.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:26.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:26.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:26.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:26.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:26.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:26.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:27.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:27.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:27.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:27.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:28.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:28.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:28.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:28.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:29.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:29.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:29.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:29.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:30.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:30.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:30.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:30.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:31.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:31.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:31.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:31.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:32.947 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:32.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:33.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:33.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:33.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:33.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:34.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:34.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:34.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:34.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:35.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:35.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:35.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:35.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:36.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:36.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:36.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:36.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:37.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:37.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:37.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:37.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:37.948 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:37.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:38.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:38.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:38.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:38.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:39.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:39.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:39.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:39.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:40.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:40.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:40.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:40.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:41.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:41.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:41.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:41.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:42.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:42.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:42.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:42.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:42.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:42.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:43.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:43.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:43.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:43.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:44.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:44.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:44.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:44.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:45.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:45.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:45.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:45.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:46.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:46.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:46.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:46.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:47.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:47.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:47.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:47.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:47.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:47.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:48.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:48.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:49.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:49.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:49.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:49.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:50.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:50.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:50.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:50.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:51.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:51.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:51.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:51.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:52.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:52.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:52.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:52.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:52.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:52.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:53.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:53.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:53.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:53.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:54.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:54.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:54.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:54.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:55.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:55.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:55.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:55.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:56.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:56.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:56.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:56.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:57.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:57.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:57.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:57.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:57.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:34:57.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:34:58.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:58.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:58.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:58.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:34:59.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:59.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:34:59.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:34:59.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:00.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:00.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:00.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:00.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:01.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:01.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:01.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:01.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:02.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:02.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:02.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:02.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:02.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:02.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:03.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:03.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:03.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:03.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:04.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:04.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:04.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:04.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:05.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:05.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:06.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:06.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:06.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:06.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:07.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:07.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:07.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:07.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:07.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:07.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:08.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:08.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:08.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:08.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:09.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:09.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:09.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:09.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:10.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:10.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:10.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:10.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:11.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:11.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:11.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:11.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:12.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:12.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:12.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:12.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:12.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:12.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:13.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:13.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:13.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:13.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:14.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:14.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:14.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:14.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:15.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:15.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:15.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:15.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:16.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:16.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:16.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:16.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:17.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:17.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:17.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:17.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:17.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:17.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:18.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:18.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:18.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:18.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:19.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:19.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:19.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:19.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:20.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:20.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:20.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:20.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:21.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:21.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:21.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:21.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:22.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:22.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:22.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:22.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:22.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:22.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:23.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:23.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:23.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:23.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:24.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:24.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:25.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:25.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:25.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:25.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:26.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:26.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:26.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:26.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:27.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:27.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:27.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:27.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:27.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:27.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:28.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:28.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:28.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:28.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:29.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:29.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:29.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:29.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:30.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:30.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:30.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:30.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:30.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:30.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:30.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:30.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:31.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:31.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:31.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:31.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:32.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:32.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:33.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:33.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:33.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:33.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:33.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:33.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:33.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:33.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:34.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:34.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:35.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:35.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:35.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:35.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:36.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:36.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:36.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:36.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:37.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:37.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:37.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:37.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:37.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:37.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:38.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:38.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:38.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:38.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:40.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:40.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:40.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:40.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:41.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:41.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:41.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:41.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:42.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:42.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:42.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:42.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:42.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:42.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:43.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:43.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:44.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:44.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:44.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:44.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:45.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:45.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:45.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:45.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:46.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:46.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:46.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:46.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:47.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:47.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:47.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:47.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:47.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:47.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:48.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:48.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:48.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:48.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:48.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:48.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:48.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:48.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:49.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:49.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:49.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:49.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:50.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:50.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:50.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:50.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:51.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:51.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:51.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:51.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:52.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:52.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:52.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:52.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:52.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:52.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:53.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:53.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:53.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:53.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:54.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:54.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:54.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:54.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:55.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:55.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:55.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:55.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:56.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:56.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:56.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:56.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:57.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:57.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:57.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:35:57.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:35:58.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:58.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:58.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:58.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:35:59.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:59.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:35:59.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:35:59.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:00.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:00.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:00.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:00.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:01.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:01.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:01.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:01.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:02.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:02.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:02.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:02.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:02.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:02.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:03.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:03.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:03.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:03.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:04.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:04.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:04.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:04.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:05.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:05.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:05.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:05.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:06.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:06.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:06.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:06.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:07.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:07.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:07.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:07.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:07.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:07.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:08.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:08.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:08.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:08.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:09.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:09.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:09.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:09.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:10.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:10.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:10.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:10.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:11.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:11.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:11.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:11.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:12.966 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:12.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:12.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:12.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:12.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:12.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:14.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:13.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:14.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:13.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:14.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:14.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:15.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:15.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:15.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:15.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:17.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:16.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:17.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:16.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:17.969 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:17.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:17.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:17.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:17.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:17.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:18.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:18.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:18.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:18.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:19.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:19.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:19.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:19.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:20.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:20.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:20.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:20.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:21.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:21.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:21.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:21.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:22.969 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:22.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:23.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:23.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:23.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:23.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:24.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:24.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:24.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:24.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:24.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:24.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:24.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:24.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:25.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:25.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:25.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:25.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:26.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:26.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:26.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:26.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:27.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:27.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:27.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:27.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:27.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:27.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:28.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:28.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:28.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:28.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:29.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:29.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:29.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:29.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:30.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:30.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:30.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:30.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:31.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:31.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:31.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:31.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:32.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:32.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:32.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:32.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:32.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:32.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:33.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:33.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:33.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:33.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:34.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:34.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:34.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:34.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:35.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:35.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:35.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:35.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:36.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:36.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:36.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:36.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:37.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:37.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:37.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:37.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:37.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:37.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:38.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:38.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:38.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:38.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:39.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:39.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:39.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:40.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:40.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:40.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:40.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:41.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:41.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:41.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:41.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:42.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:42.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:42.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:42.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:42.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:42.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:43.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:43.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:43.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:43.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:44.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:44.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:44.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:44.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:45.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:45.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:45.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:45.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:46.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:46.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:46.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:46.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:47.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:47.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:47.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:47.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:47.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:47.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:48.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:48.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:48.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:48.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:49.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:49.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:49.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:49.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:50.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:50.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:50.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:50.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:51.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:51.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:51.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:51.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:52.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:52.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:52.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:52.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:52.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:52.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:54.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:53.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:54.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:53.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:55.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:54.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:55.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:54.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:56.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:56.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:56.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:56.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:57.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:57.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:57.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:57.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:57.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:36:57.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:36:58.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:58.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:58.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:58.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:36:59.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:59.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:36:59.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:36:59.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:00.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:00.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:00.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:00.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:01.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:01.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:01.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:01.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:02.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:02.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:02.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:02.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:02.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:02.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:03.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:03.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:03.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:03.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:04.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:04.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:04.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:04.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:05.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:05.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:05.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:05.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:06.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:06.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:06.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:06.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:07.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:07.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:07.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:07.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:07.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:07.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:08.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:08.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:08.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:08.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:09.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:09.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:09.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:09.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:10.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:10.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:10.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:10.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:11.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:11.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:11.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:11.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:12.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:12.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:12.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:12.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:13.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:13.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:13.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:13.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:14.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:14.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:14.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:14.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:15.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:15.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:15.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:15.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:16.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:16.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:16.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:16.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:17.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:17.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:17.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:17.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:17.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:17.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:18.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:18.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:18.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:18.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:19.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:19.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:19.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:19.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:20.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:20.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:20.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:20.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:21.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:21.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:21.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:21.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:22.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:22.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:22.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:22.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:22.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:22.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:23.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:23.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:23.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:23.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:24.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:24.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:24.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:24.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:25.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:25.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:25.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:25.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:26.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:26.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:26.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:26.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:27.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:27.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:27.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:27.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:27.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:27.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:28.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:28.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:28.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:28.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:29.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:29.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:29.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:29.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:30.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:30.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:30.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:30.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:31.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:31.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:31.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:31.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:32.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:32.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:32.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:32.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:32.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:32.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:33.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:33.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:33.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:33.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:34.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:34.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:34.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:34.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:35.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:35.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:35.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:35.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:36.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:36.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:36.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:36.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:37.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:37.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:37.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:37.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:37.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:37.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:38.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:38.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:38.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:38.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:39.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:39.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:39.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:39.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:40.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:40.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:40.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:40.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:41.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:41.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:41.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:41.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:42.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:42.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:42.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:42.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:42.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:42.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:43.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:43.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:43.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:43.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:44.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:44.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:44.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:44.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:45.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:45.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:45.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:45.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:46.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:46.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:46.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:46.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:47.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:47.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:47.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:47.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:47.989 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:47.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:48.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:48.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:48.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:48.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:49.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:49.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:49.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:49.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:50.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:50.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:50.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:50.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:51.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:51.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:51.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:51.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:52.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:52.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:52.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:52.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:52.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:52.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:53.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:53.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:53.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:53.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:54.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:54.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:54.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:54.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:55.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:55.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:55.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:55.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:56.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:56.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:56.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:56.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:57.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:57.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:57.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:37:57.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:37:58.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:58.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:58.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:58.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:37:59.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:59.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:37:59.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:37:59.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:00.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:00.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:00.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:00.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:01.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:01.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:01.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:01.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:02.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:02.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:02.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:02.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:02.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:02.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:03.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:03.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:03.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:03.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:04.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:04.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:04.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:04.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:05.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:05.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:05.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:05.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:06.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:06.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:06.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:06.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:07.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:07.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:07.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:07.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:07.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:08.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:08.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:08.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:08.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:09.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:09.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:09.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:09.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:10.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:10.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:10.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:10.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:11.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:11.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:11.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:11.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:12.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:12.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:12.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:12.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:12.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:12.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:13.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:13.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:13.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:13.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:14.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:14.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:14.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:14.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:15.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:15.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:15.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:15.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:16.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:16.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:16.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:16.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:17.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:17.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:17.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:17.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:17.996 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:17.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:18.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:18.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:18.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:18.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:19.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:19.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:19.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:19.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:20.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:20.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:20.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:20.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:21.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:21.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:21.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:21.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:22.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:22.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:22.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:22.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:22.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:22.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:23.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:23.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:23.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:23.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:24.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:24.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:25.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:25.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:25.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:25.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:26.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:26.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:26.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:26.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:27.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:27.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:27.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:27.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:27.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:27.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:28.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:28.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:28.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:28.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:29.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:29.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:29.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:29.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:30.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:30.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:30.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:30.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:31.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:31.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:31.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:31.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:32.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:32.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:32.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:32.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:32.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:32.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:33.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:33.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:33.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:33.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:34.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:34.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:34.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:34.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:35.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:35.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:35.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:35.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:36.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:36.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:36.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:36.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:37.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:37.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:37.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:37.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:37.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:37.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:38.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:38.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:38.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:38.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:39.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:39.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:39.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:39.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:40.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:40.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:41.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:41.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:41.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:41.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:42.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:42.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:42.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:42.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:43.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:42.995+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:43.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:43.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:43.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:43.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:44.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:44.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:44.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:44.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:45.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:45.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:45.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:45.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:46.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:46.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:46.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:46.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:47.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:47.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:47.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:47.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:48.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:47.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:48.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:48.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:48.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:48.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:49.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:49.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:49.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:49.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:50.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:50.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:50.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:50.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:51.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:51.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:51.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:51.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:52.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:52.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:52.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:52.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:53.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:52.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:53.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:53.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:53.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:53.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:54.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:54.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:54.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:54.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:55.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:55.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:55.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:55.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:56.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:56.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:56.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:56.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:57.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:57.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:57.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:57.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:58.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:38:57.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:38:58.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:58.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:58.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:58.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:38:59.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:59.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:38:59.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:38:59.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:00.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:00.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:00.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:00.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:01.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:01.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:01.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:01.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:02.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:02.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:02.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:02.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:03.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:03.002+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:03.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:03.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:03.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:03.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:04.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:04.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:04.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:04.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:05.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:05.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:05.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:06.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:06.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:06.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:06.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:07.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:07.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:07.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:07.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:08.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:08.002+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:08.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:08.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:08.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:08.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:09.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:09.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:09.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:09.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:10.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:10.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:10.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:10.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:11.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:11.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:11.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:11.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:12.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:12.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:12.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:12.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:13.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:13.002+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:13.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:13.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:13.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:13.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:14.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:14.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:14.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:14.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:15.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:15.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:15.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:15.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:16.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:16.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:16.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:16.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:17.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:17.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:17.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:17.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:18.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:18.006+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:18.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:18.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:18.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:18.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:19.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:19.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:19.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:19.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:20.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:20.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:20.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:20.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:21.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:21.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:21.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:21.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:22.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:22.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:22.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:22.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:23.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:23.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:23.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:23.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:23.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:23.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:24.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:24.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:24.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:24.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:25.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:25.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:25.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:25.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:26.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:26.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:26.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:26.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:27.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:27.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:27.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:27.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:28.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:28.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:28.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:28.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:28.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:28.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:29.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:29.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:29.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:29.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:30.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:30.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:30.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:30.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:31.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:31.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:31.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:31.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:32.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:32.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:32.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:32.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:33.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:33.009+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:33.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:33.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:33.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:33.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:34.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:34.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:34.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:34.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:35.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:35.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:35.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:35.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:36.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:36.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:36.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:36.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:37.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:37.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:37.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:37.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:38.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:38.009+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:38.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:38.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:39.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:39.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:39.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:39.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:40.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:40.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:40.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:40.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:41.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:41.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:41.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:41.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:42.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:42.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:42.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:42.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:43.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:43.013+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:43.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:43.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:43.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:43.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:44.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:44.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:44.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:44.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:45.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:45.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:46.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:46.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:46.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:46.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:47.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:47.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:47.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:47.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:48.019 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:48.013+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:48.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:48.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:48.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:48.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:49.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:49.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:49.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:49.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:50.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:50.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:50.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:50.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:51.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:51.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:51.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:51.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:52.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:52.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:52.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:52.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:53.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:53.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:53.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:53.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:53.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:53.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:54.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:54.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:54.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:54.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:55.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:55.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:55.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:55.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:56.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:56.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:56.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:56.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:57.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:57.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:57.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:57.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:58.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:39:58.016+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:39:58.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:58.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:58.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:58.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:39:59.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:59.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:39:59.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:39:59.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:00.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:00.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:00.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:00.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:01.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:01.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:01.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:01.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:02.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:02.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:02.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:02.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:03.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:03.016+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:03.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:03.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:03.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:03.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:04.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:04.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:04.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:04.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:05.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:05.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:05.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:05.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:06.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:06.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:06.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:06.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:07.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:07.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:07.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:07.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:08.022 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:08.016+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:08.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:08.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:08.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:08.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:09.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:09.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:09.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:09.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:10.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:10.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:11.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:11.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:11.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:11.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:12.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:12.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:12.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:12.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:13.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:13.016+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:13.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:13.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:13.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:13.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:14.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:14.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:14.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:14.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:15.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:15.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:15.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:15.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:16.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:16.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:16.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:16.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:17.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:17.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:17.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:17.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:18.025 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:18.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:18.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:18.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:18.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:18.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:19.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:19.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:19.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:19.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:20.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:20.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:20.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:20.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:21.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:21.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:21.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:21.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:22.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:22.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:22.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:22.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:23.025 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:23.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:23.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:23.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:23.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:23.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:24.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:24.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:24.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:24.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:25.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:25.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:25.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:25.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:26.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:26.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:26.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:26.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:27.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:27.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:27.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:27.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:28.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:28.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:28.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:28.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:28.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:28.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:29.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:29.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:29.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:29.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:30.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:30.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:30.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:30.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:31.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:31.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:31.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:31.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:31.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:31.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:31.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:31.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:33.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:33.023+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:33.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:33.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:33.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:33.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:34.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:34.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:34.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:34.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:35.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:34.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:35.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:34.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:36.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:36.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:36.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:36.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:37.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:37.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:37.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:37.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:37.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:37.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:37.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:37.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:38.029 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:38.023+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:39.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:39.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:39.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:39.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:40.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:40.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:40.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:40.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:41.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:41.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:41.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:41.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:42.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:42.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:42.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:42.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:43.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:43.023+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:43.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:43.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:43.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:43.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:44.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:44.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:44.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:44.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:45.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:45.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:45.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:45.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:46.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:46.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:46.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:46.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:47.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:47.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:47.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:47.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:48.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:48.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:48.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:48.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:48.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:48.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:49.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:49.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:49.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:49.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:50.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:50.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:50.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:50.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:50.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:50.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:50.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:50.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:51.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:51.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:51.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:51.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:53.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:52.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:53.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:52.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:53.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:53.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:53.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:53.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:53.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:53.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:54.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:54.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:54.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:54.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:56.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:56.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:56.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:56.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:56.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:56.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:56.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:56.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:57.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:57.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:57.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:57.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:58.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:40:58.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:40:58.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:58.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:58.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:58.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:40:59.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:59.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:40:59.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:40:59.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:00.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:00.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:01.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:01.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:01.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:01.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:02.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:02.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:02.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:02.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:03.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:03.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:03.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:03.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:03.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:03.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:04.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:04.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:04.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:04.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:05.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:05.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:05.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:05.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:06.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:06.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:06.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:06.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:07.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:07.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:07.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:07.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:08.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:08.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:08.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:08.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:08.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:08.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:09.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:09.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:10.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:09.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:10.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:10.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:10.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:10.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:11.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:11.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:11.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:11.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:13.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:13.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:13.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:13.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:13.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:13.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:13.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:13.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:14.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:14.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:14.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:14.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:16.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:16.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:16.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:16.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:16.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:16.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:16.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:16.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:17.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:17.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:17.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:17.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:18.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:18.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:19.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:19.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:19.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:19.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:20.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:20.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:20.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:20.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:21.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:21.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:22.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:22.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:22.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:22.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:23.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:23.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:23.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:23.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:23.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:23.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:24.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:24.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:24.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:24.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:25.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:25.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:25.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:25.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:26.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:26.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:26.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:26.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:27.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:27.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:27.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:27.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:28.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:28.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:28.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:28.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:28.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:28.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:29.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:29.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:29.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:29.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:30.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:30.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:30.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:30.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:31.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:31.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:31.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:31.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:32.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:32.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:32.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:32.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:33.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:33.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:33.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:33.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:33.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:33.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:34.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:34.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:34.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:34.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:35.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:35.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:35.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:35.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:36.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:36.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:36.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:36.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:37.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:37.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:37.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:37.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:38.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:38.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:38.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:38.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:38.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:38.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:39.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:39.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:39.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:39.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:40.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:40.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:40.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:40.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:41.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:41.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:41.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:41.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:42.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:42.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:42.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:42.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:43.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:43.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:43.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:43.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:43.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:44.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:44.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:44.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:44.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:45.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:45.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:45.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:45.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:46.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:46.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:46.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:46.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:47.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:47.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:47.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:47.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:48.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:48.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:48.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:48.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:48.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:48.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:49.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:49.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:49.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:49.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:50.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:50.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:51.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:51.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:51.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:51.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:52.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:52.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:52.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:52.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:53.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:53.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:53.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:53.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:53.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:53.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:54.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:54.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:54.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:54.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:55.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:55.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:56.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:56.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:56.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:56.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:57.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:57.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:57.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:57.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:58.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:41:58.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:41:58.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:58.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:58.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:41:59.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:59.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:41:59.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:41:59.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:00.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:00.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:01.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:01.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:01.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:01.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:02.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:02.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:02.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:02.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:03.051 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:03.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:03.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:03.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:03.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:03.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:04.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:04.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:04.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:04.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:05.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:05.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:05.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:05.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:06.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:06.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:06.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:06.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:07.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:07.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:07.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:07.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:08.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:08.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:08.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:08.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:08.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:08.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:09.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:09.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:09.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:09.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:10.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:10.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:10.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:10.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:11.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:11.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:11.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:11.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:12.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:12.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:12.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:12.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:13.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:13.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:13.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:13.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:13.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:13.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:14.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:14.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:14.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:14.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:15.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:15.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:15.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:15.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:16.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:16.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:16.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:16.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:17.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:17.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:17.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:17.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:18.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:18.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:18.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:18.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:18.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:18.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:19.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:19.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:19.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:19.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:20.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:20.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:20.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:20.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:21.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:21.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:21.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:21.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:22.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:22.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:22.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:22.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:23.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:23.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:23.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:23.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:23.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:23.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:24.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:24.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:24.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:24.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:25.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:25.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:25.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:25.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:26.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:26.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:26.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:26.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:27.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:27.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:27.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:27.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:28.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:28.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:28.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:28.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:28.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:28.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:29.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:29.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:29.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:29.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:30.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:30.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:30.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:30.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:31.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:31.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:31.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:31.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:32.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:32.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:32.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:32.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:33.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:33.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:33.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:33.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:33.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:33.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:34.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:34.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:34.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:34.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:35.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:35.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:35.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:35.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:36.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:36.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:37.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:37.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:37.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:37.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:38.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:38.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:38.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:38.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:38.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:38.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:39.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:39.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:39.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:39.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:40.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:40.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:40.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:40.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:41.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:41.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:41.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:41.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:42.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:42.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:42.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:42.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:43.059 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:43.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:43.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:43.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:44.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:44.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:44.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:44.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:45.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:45.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:45.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:45.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:46.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:46.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:46.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:46.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:47.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:47.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:47.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:47.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:48.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:48.056+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:48.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:48.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:48.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:48.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:49.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:49.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:49.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:49.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:50.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:50.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:50.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:50.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:51.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:51.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:51.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:51.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:52.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:52.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:52.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:52.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:53.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:53.056+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:53.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:53.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:53.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:53.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:54.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:54.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:54.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:54.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:55.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:55.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:55.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:55.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:56.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:56.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:56.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:56.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:57.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:57.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:57.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:57.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:58.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:42:58.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:42:58.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:58.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:58.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:58.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:42:59.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:59.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:42:59.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:42:59.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:00.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:00.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:00.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:00.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:01.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:01.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:01.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:01.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:02.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:02.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:02.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:02.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:03.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:03.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:03.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:03.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:03.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:03.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:04.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:04.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:04.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:04.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:05.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:05.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:05.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:05.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:06.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:06.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:06.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:06.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:07.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:07.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:07.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:07.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:08.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:08.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:08.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:08.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:08.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:08.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:09.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:09.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:09.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:09.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:10.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:10.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:10.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:10.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:11.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:11.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:11.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:11.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:12.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:12.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:32:40.880484+0000 (oldest deadline 2019-10-17T21:33:00.880484+0000) 2019-10-17T21:43:12.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:12.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:13.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:13.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:13.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:13.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:14.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:14.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:15.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:15.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:16.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:16.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:17.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:17.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 36 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:18.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:18.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:18.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:18.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:19.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:19.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:20.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:20.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:21.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:21.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:22.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:22.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:23.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:23.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:23.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:23.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:24.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:24.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:25.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:25.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:26.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:26.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:27.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:26.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:28.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:28.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:28.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:28.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:29.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:29.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:30.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:30.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:31.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:31.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:32.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:32.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:33.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:33.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:33.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:33.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:34.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:34.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:35.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:35.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:35.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:35.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:36.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:36.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:36.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:36.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:37.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:37.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:37.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:37.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:37.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:37.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:37.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:37.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:38.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:38.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:38.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:38.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:38.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:38.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:40.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:39.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:40.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:39.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:41.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:41.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:41.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:41.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:42.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:42.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:42.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:42.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:42.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:42.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:42.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:42.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:43.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:43.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:43.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:43.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:43.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:43.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:45.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:44.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:45.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:44.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:45.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:45.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:45.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:45.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:46.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:46.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:46.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:46.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:47.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:47.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:47.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:47.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:48.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:48.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:48.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:48.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:48.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:48.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:49.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:49.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:49.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:49.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:50.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:50.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:50.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:50.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:51.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:51.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:51.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:51.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:53.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:52.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:53.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:52.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:53.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:53.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:53.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:53.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:53.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:53.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:55.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:55.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:55.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:55.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:56.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:56.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:56.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:56.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:56.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:57.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:56.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:57.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:57.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:57.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:57.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:58.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:43:58.066+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:43:58.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:58.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:58.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:58.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:43:59.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:59.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:43:59.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:43:59.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:00.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:00.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:00.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:00.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:01.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:01.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:01.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:01.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:02.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:02.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:02.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:02.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:03.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:03.066+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:03.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:03.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:03.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:03.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:04.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:04.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:04.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:04.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:05.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:05.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:05.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:05.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:06.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:06.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:06.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:06.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:07.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:07.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:07.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:07.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:08.076 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:08.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:08.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:08.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:08.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:08.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:09.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:09.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:09.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:09.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:10.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:10.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:10.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:10.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:11.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:11.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:11.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:11.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:12.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:12.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:12.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:12.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:13.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:13.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:13.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:13.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:13.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:13.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:14.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:14.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:15.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:15.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:15.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:15.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:16.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:16.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:16.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:16.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:17.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:17.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:17.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:17.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:18.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:18.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:18.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:18.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:18.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:18.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:19.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:19.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:19.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:19.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:20.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:20.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:20.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:20.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:21.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:21.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:21.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:21.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:22.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:22.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:22.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:22.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:23.080 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:23.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:23.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:23.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:23.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:23.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:24.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:24.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:24.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:24.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:25.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:25.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:25.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:25.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:26.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:26.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:27.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:27.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:27.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:27.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:28.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:28.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:28.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:28.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:28.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:28.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:29.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:29.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:29.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:29.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:30.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:30.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:30.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:30.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:31.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:31.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:31.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:31.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:32.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:32.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:32.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:32.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:33.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:33.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:33.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:33.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:33.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:33.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:34.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:34.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:34.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:34.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:35.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:35.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:35.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:35.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:36.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:36.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:36.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:36.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:37.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:37.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:37.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:37.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:38.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:38.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:38.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:38.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:38.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:38.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:39.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:39.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:39.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:39.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:40.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:40.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:40.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:40.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:41.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:41.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:41.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:41.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:42.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:42.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:42.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:42.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:43.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:43.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:43.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:43.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:43.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:43.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:44.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:44.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:44.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:44.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:45.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:45.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:45.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:45.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:46.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:46.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:46.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:46.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:47.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:47.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:47.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:47.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:48.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:48.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:48.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:48.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:49.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:49.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:49.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:49.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:50.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:50.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:50.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:50.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:51.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:51.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:51.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:51.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:52.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:52.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:52.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:52.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:53.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:53.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:53.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:53.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:53.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:53.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:54.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:54.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:54.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:54.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:55.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:55.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:55.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:55.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:56.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:56.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:56.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:56.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:57.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:57.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:57.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:57.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:58.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:44:58.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:44:58.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:58.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:58.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:58.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:44:59.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:59.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:44:59.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:44:59.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:00.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:00.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:00.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:00.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:01.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:01.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:01.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:01.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:02.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:02.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:02.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:02.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:03.088 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:03.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:03.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:03.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:03.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:03.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:04.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:04.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:04.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:04.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:05.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:05.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:05.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:05.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:06.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:06.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:06.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:06.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:07.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:07.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:07.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:07.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:08.088 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:08.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:08.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:08.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:08.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:08.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:09.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:09.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:09.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:09.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:10.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:10.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:10.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:10.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:11.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:11.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:11.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:11.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:12.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:12.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:12.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:12.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:13.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:13.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:13.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:13.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:13.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:13.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:14.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:14.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:14.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:14.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:15.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:15.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:15.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:15.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:16.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:16.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:16.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:16.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:17.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:17.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:17.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:17.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:18.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:18.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:18.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:18.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:18.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:18.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:19.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:19.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:20.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:20.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:20.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:20.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:21.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:21.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:21.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:21.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:22.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:22.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:22.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:22.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:23.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:23.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:23.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:23.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:23.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:23.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:24.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:24.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:24.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:24.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:25.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:25.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:25.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:26.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:26.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:26.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:26.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:27.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:27.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:27.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:27.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:28.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:28.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:28.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:28.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:28.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:28.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:29.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:29.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:29.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:29.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:30.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:30.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:30.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:30.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:31.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:31.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:31.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:31.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:32.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:32.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:32.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:32.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:33.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:33.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:33.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:33.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:33.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:33.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:34.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:34.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:34.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:34.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:35.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:35.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:35.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:35.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:36.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:36.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:36.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:36.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:37.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:37.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:37.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:37.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:38.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:38.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:38.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:38.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:38.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:38.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:39.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:39.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:39.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:39.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:40.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:40.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:40.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:40.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:41.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:41.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:41.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:41.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:42.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:42.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:42.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:42.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:43.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:43.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:43.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:43.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:43.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:43.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:44.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:44.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:45.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:45.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:45.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:45.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:46.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:46.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:46.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:46.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:47.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:47.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:47.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:47.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:48.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:48.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:48.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:48.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:48.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:48.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:49.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:49.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:49.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:49.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:50.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:50.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:50.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:50.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:51.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:51.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:51.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:51.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:52.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:52.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:52.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:52.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:53.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:53.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:53.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:53.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:53.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:53.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:54.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:54.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:54.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:54.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:55.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:55.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:55.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:55.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:56.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:56.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:56.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:56.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:57.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:57.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:57.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:57.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:58.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:45:58.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:45:58.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:58.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:58.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:58.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:45:59.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:59.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:45:59.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:45:59.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:00.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:00.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:00.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:00.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:01.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:01.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:01.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:01.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:02.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:02.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:02.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:02.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:03.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:03.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:03.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:03.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:03.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:03.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:04.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:04.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:05.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:05.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:05.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:05.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:06.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:06.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:06.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:06.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:07.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:07.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:07.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:07.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:08.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:08.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:08.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:08.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:08.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:08.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:09.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:09.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:09.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:09.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:10.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:10.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:10.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:10.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:11.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:11.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:11.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:11.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:12.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:12.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:12.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:12.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:13.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:13.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:13.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:13.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:13.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:13.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:14.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:14.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:14.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:14.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:15.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:15.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:15.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:15.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:16.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:16.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:16.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:16.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:17.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:17.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:17.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:17.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:18.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:18.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:18.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:18.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:18.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:18.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:19.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:19.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:19.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:19.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:20.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:20.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:20.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:20.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:21.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:21.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:21.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:21.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:22.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:22.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:22.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:22.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:23.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:23.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:23.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:23.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:23.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:23.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:24.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:24.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:24.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:24.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:25.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:25.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:25.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:25.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:26.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:26.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:26.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:26.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:27.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:27.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:27.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:27.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:28.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:28.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:28.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:28.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:28.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:28.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:29.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:29.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:29.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:29.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:30.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:30.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:30.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:30.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:31.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:31.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:31.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:31.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:32.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:32.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:32.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:32.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:33.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:33.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:33.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:33.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:33.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:33.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:34.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:34.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:34.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:34.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:35.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:35.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:35.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:35.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:36.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:36.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:37.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:37.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:37.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:37.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:38.112 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:38.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:38.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:38.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:39.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:39.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:39.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:39.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:40.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:40.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:40.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:40.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:41.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:41.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:41.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:41.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:42.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:42.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:42.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:42.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:43.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:43.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:43.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:43.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:43.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:43.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:44.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:44.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:44.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:44.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:45.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:45.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:45.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:45.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:46.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:46.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:46.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:46.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:47.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:47.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:47.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:47.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:48.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:48.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:48.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:48.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:48.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:48.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:49.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:49.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:49.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:49.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:50.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:50.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:50.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:50.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:51.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:51.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:51.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:51.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:52.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:52.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:52.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:52.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:53.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:53.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:53.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:53.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:53.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:53.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:54.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:54.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:54.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:54.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:55.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:55.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:55.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:55.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:56.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:56.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:56.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:56.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:57.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:57.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:57.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:57.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:58.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:46:58.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:46:58.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:58.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:58.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:58.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:46:59.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:59.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:46:59.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:46:59.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:00.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:00.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:00.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:00.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:01.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:01.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:01.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:01.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:02.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:02.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:02.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:02.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:03.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:03.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:03.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:03.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:03.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:03.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:04.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:04.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:04.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:04.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:05.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:05.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:05.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:05.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:06.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:06.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:06.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:06.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:07.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:07.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:07.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:07.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:08.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:08.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:08.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:08.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:08.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:08.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:09.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:09.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:09.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:09.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:10.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:10.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:10.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:10.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:11.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:11.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:11.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:11.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:12.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:12.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:12.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:12.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:13.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:13.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:13.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:13.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:13.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:13.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:14.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:14.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:14.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:14.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:15.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:15.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:15.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:15.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:16.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:16.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:16.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:16.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:17.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:17.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:17.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:17.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:18.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:18.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:18.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:18.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:19.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:19.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:19.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:19.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:20.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:20.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:20.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:20.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:21.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:21.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:21.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:21.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:22.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:22.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:22.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:22.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:23.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:23.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:23.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:23.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:23.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:23.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:24.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:24.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:25.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:25.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:25.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:25.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:26.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:26.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:26.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:26.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:27.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:27.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:27.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:27.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:28.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:28.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:28.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:28.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:28.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:28.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:29.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:29.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:29.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:29.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:30.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:30.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:30.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:30.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:31.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:31.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:31.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:31.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:32.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:32.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:32.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:32.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:33.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:33.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:33.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:33.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:33.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:33.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:34.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:34.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:34.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:34.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:35.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:35.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:35.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:35.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:36.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:36.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:36.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:36.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:37.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:37.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:37.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:37.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:38.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:38.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:38.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:38.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:38.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:38.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:39.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:39.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:39.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:39.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:40.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:40.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:40.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:40.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:41.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:41.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:41.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:41.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:42.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:42.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:42.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:42.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:43.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:43.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:43.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:43.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:43.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:43.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:44.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:44.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:44.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:44.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:45.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:45.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:45.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:45.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:46.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:46.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:46.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:46.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:47.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:47.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:47.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:47.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:48.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:48.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:48.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:48.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:48.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:48.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:49.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:49.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:49.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:49.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:50.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:50.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:50.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:50.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:51.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:51.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:51.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:51.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:52.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:52.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:52.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:52.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:53.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:53.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:53.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:53.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:53.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:53.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:54.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:54.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:55.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:55.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:55.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:55.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:56.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:56.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:56.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:56.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:57.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:57.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:57.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:57.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:58.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:47:58.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:47:58.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:58.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:58.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:58.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:47:59.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:59.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:47:59.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:47:59.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:00.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:00.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:00.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:00.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:01.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:01.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:01.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:01.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:02.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:02.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:02.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:02.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:03.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:03.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:03.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:03.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:03.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:03.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:04.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:04.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:04.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:04.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:05.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:05.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:05.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:05.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:06.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:06.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:06.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:06.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:07.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:07.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:07.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:07.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:07.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:07.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:07.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:07.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:08.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:08.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:09.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:08.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:09.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:08.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:10.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:10.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:10.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:10.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:11.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:11.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:11.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:11.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:12.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:12.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:12.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:12.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:13.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:13.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:13.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:13.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:13.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:13.126+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:14.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:14.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:14.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:14.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:15.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:15.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:15.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:15.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:16.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:16.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:16.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:16.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:17.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:17.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:17.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:17.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:18.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:18.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:18.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:18.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:18.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:18.126+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:19.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:19.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:19.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:19.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:20.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:20.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:20.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:20.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:21.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:21.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:21.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:21.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:22.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:22.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:22.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:22.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:23.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:23.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:23.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:23.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:23.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:23.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:23.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:23.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:23.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:24.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:24.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:24.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:24.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:25.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:25.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:25.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:25.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:26.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:26.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:26.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:26.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:27.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:27.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:27.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:27.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:28.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:28.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:28.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:28.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:28.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:28.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:29.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:29.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:29.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:29.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:30.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:30.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:30.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:30.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:31.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:31.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:31.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:31.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:32.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:32.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:32.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:32.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:33.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:33.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:33.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:33.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:33.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:33.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:34.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:34.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:34.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:34.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:35.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:35.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:35.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:35.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:36.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:36.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:36.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:36.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:37.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:37.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:37.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:37.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:38.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:38.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:38.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:38.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:38.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:38.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:39.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:39.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:39.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:39.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:40.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:40.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:40.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:40.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:41.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:41.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:41.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:41.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:42.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:42.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:42.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:42.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:43.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:43.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:43.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:43.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:43.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:43.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:44.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:44.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:44.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:44.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:45.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:45.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:45.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:45.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:46.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:46.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:46.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:46.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:47.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:47.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:47.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:47.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:48.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:48.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:48.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:48.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:48.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:48.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:49.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:49.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:49.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:49.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:50.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:50.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:50.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:50.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:51.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:51.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:51.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:51.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:52.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:52.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:52.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:52.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:53.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:53.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:53.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:53.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:53.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:53.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:54.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:54.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:54.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:54.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:55.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:55.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:55.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:55.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:56.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:56.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:56.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:56.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:57.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:57.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:57.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:57.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:58.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:48:58.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:48:58.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:58.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:58.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:58.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:48:59.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:59.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:48:59.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:48:59.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:01.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:00.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:01.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:00.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:01.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:01.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:01.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:01.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:02.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:02.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:02.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:02.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:03.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:03.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:04.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:04.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:04.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:04.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:04.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:04.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:04.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:04.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:05.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:05.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:05.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:05.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:06.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:06.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:06.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:06.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:07.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:07.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:07.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:07.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:08.151 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:08.137+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:08.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:08.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:08.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:08.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:09.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:09.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:09.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:09.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:10.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:10.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:10.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:10.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:11.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:11.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:11.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:11.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:13.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:12.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:13.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:12.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:13.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:13.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:14.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:14.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:14.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:14.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:15.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:15.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:15.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:15.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:16.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:16.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:16.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:16.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:17.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:17.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:17.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:17.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:18.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:18.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:18.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:18.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:18.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:18.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:19.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:19.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:19.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:19.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:20.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:20.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:20.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:20.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:21.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:21.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:21.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:21.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:22.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:22.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:22.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:22.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:22.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:22.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:22.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:22.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:23.146 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:23.140+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:23.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:23.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:23.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:23.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:24.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:24.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:24.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:24.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:25.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:25.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:25.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:25.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:26.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:26.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:26.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:26.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:27.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:27.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:27.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:27.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:28.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:28.140+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:28.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:28.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:28.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:28.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:29.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:29.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:29.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:29.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:30.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:30.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:30.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:30.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:31.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:31.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:31.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:31.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:32.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:32.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:32.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:32.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:33.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:33.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:33.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:33.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:33.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:33.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:34.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:34.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:34.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:34.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:35.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:35.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:35.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:35.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:37.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:37.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:37.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:37.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:38.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:38.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:38.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:38.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:38.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:38.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:39.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:39.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:39.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:39.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:40.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:40.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:40.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:40.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:41.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:41.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:41.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:41.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:42.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:42.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:42.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:42.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:43.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:43.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:43.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:43.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:43.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:43.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:44.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:44.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:44.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:44.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:45.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:45.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:45.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:45.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:46.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:46.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:46.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:46.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:47.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:47.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:47.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:47.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:48.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:48.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:48.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:48.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:48.150 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:48.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:49.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:49.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:49.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:50.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:50.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:50.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:50.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:51.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:51.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:51.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:51.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:52.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:52.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:52.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:52.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:53.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:53.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:53.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:53.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:53.151 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:53.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:54.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:54.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:54.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:54.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:55.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:55.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:55.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:55.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:56.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:56.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:56.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:56.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:57.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:57.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:57.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:57.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:58.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:58.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:58.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:49:58.151 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:49:58.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:49:59.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:59.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:49:59.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:49:59.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:00.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:00.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:00.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:00.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:01.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:01.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:01.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:01.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:02.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:02.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:02.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:02.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:03.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:02.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:03.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:02.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:03.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:03.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:04.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:04.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:04.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:04.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:05.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:05.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:05.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:05.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:05.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:05.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:05.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:05.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:06.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:06.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:06.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:06.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:07.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:07.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:07.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:07.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:08.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:08.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:08.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:08.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:08.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:08.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:09.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:09.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:09.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:09.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:11.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:11.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:11.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:11.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:12.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:12.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:12.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:12.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:13.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:13.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:13.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:13.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:13.154 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:13.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:14.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:14.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:14.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:14.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:15.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:15.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:15.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:15.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:16.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:16.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:16.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:16.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:17.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:17.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:17.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:17.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:18.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:18.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:18.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:18.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:18.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:18.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:19.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:19.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:19.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:19.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:20.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:20.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:20.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:20.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:21.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:21.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:21.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:21.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:22.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:22.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:22.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:22.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:23.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:23.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:23.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:23.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:23.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:23.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:24.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:24.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:24.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:24.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:25.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:25.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:25.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:25.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:26.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:26.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:26.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:26.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:27.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:27.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:27.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:27.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:28.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:28.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:28.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:28.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:28.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:28.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:29.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:29.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:29.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:29.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:30.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:30.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:30.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:30.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:31.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:31.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:31.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:31.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:32.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:32.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:32.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:32.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:33.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:33.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:33.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:33.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:33.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:33.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:34.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:34.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:34.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:34.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:35.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:35.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:35.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:35.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:36.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:36.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:36.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:36.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:37.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:37.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:37.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:37.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:38.158 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:38.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:38.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:38.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:38.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:38.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:39.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:39.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:39.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:39.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:40.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:40.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:40.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:40.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:41.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:41.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:41.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:41.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:42.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:42.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:42.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:42.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:43.159 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:43.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:43.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:43.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:43.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:43.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:44.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:44.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:44.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:44.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:45.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:45.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:45.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:45.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:46.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:46.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:46.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:46.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:47.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:47.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:47.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:47.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:48.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:48.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:48.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:48.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:48.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:48.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:49.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:49.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:49.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:49.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:50.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:50.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:50.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:50.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:51.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:51.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:51.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:51.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:52.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:52.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:52.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:52.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:53.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:53.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:53.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:53.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:53.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:53.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:54.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:54.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:54.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:54.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:55.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:55.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:55.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:55.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:56.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:56.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:56.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:56.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:57.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:57.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:57.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:57.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:58.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:50:58.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:50:58.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:58.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:58.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:58.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:50:59.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:59.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:50:59.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:50:59.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:00.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:00.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:00.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:00.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:01.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:01.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:01.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:01.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:02.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:02.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:02.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:02.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:03.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:03.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:03.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:03.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:03.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:03.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:04.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:04.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:04.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:04.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:05.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:05.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:05.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:06.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:06.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:06.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:07.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:07.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:07.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:07.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:08.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:08.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:08.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:08.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:08.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:08.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:09.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:09.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:09.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:09.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:10.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:10.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:10.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:10.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:11.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:11.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:11.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:11.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:12.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:12.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:12.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:12.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:13.164 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:13.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:13.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:13.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:13.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:13.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:14.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:14.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:14.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:14.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:15.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:15.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:15.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:15.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:16.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:16.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:16.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:16.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:17.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:17.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:17.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:17.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:18.164 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:18.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:18.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:18.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:18.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:18.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:19.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:19.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:19.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:19.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:20.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:20.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:20.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:20.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:21.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:21.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:21.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:21.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:22.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:22.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:22.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:22.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:23.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:23.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:23.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:23.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:23.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:23.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:24.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:24.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:24.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:24.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:25.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:25.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:25.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:25.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:26.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:26.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:26.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:26.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:27.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:27.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:27.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:27.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:28.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:28.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:28.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:28.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:28.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:28.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:29.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:29.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:29.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:29.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:30.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:30.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:30.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:30.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:31.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:31.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:31.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:31.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:32.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:32.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:32.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:32.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:33.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:33.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:33.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:33.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:33.168 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:33.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:34.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:34.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:35.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:35.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:35.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:35.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:36.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:36.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:36.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:36.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:37.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:37.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:37.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:37.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:38.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:38.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:38.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:38.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:38.168 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:38.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:39.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:39.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:39.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:39.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:40.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:40.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:40.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:40.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:41.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:41.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:41.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:41.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:41.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:41.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:41.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:41.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:42.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:42.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:42.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:42.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:43.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:43.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:43.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:43.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:43.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:43.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:45.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:45.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:45.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:45.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:46.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:46.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:46.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:46.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:47.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:47.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:47.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:47.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:48.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:48.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:48.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:48.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:48.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:48.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:49.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:48.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:49.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:48.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:50.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:50.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:50.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:50.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:51.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:51.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:51.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:51.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:52.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:52.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:52.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:52.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:53.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:53.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:53.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:53.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:53.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:53.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:54.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:54.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:54.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:54.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:54.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:54.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:54.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:54.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:56.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:56.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:56.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:56.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:57.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:57.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:57.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:57.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:58.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:58.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:58.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:58.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:51:58.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:51:58.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:51:59.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:59.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:51:59.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:51:59.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:00.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:00.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:00.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:00.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:01.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:01.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:01.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:01.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:02.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:02.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:02.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:02.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:03.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:03.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:03.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:03.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:03.171 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:03.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:04.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:04.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:04.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:04.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:05.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:05.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:05.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:05.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:06.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:06.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:06.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:06.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:07.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:07.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:07.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:07.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:08.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:08.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:08.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:08.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:08.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:08.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:09.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:09.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:09.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:09.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:10.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:10.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:10.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:10.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:11.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:11.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:11.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:11.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:12.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:12.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:12.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:12.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:13.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:13.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:13.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:13.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:13.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:13.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:14.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:14.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:14.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:14.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:15.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:15.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:15.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:15.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:16.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:16.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:16.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:16.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:17.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:17.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:17.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:17.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:18.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:18.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:18.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:18.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:18.174 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:18.167+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:19.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:19.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:19.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:19.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:20.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:20.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:20.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:20.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:21.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:21.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:21.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:21.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:22.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:22.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:22.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:22.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:22.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:22.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:23.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:23.167+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:23.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:23.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:23.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:23.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:24.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:24.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:24.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:24.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:26.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:26.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:26.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:26.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:26.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:26.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:26.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:26.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:28.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:28.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:28.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:28.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:28.176 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:28.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:28.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:28.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:28.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:28.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:29.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:29.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:29.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:29.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:30.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:30.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:30.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:30.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:31.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:31.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:31.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:31.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:32.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:32.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:32.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:32.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:33.176 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:33.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:33.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:33.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:33.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:33.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:34.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:34.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:34.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:34.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:35.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:35.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:35.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:35.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:36.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:36.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:36.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:36.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:37.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:37.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:37.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:37.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:38.179 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:38.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:38.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:38.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:38.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:38.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:39.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:39.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:39.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:39.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:40.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:40.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:40.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:40.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:41.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:41.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:41.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:41.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:42.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:42.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:42.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:42.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:43.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:43.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:43.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:43.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:43.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:43.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:44.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:44.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:44.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:44.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:45.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:45.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:45.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:45.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:46.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:46.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:46.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:46.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:47.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:47.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:47.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:47.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:48.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:48.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:48.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:48.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:48.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:48.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:49.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:49.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:49.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:49.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:50.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:50.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:50.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:50.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:51.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:51.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:51.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:51.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:52.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:52.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:52.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:52.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:53.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:53.174+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:53.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:53.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:53.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:53.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:54.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:54.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:54.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:54.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:55.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:55.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:55.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:55.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:56.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:56.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:56.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:56.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:57.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:57.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:57.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:57.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:58.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:52:58.174+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:52:58.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:58.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:58.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:58.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:52:59.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:59.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:52:59.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:52:59.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:00.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:00.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:00.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:00.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:01.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:01.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:01.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:01.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:02.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:02.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:02.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:02.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:03.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:03.174+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:03.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:03.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:03.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:03.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:04.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:04.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:04.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:04.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:05.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:05.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:05.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:05.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:06.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:06.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:06.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:06.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:07.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:07.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:07.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:07.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:08.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:08.174+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:08.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:08.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:08.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:08.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:09.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:09.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:09.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:09.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:10.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:10.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:10.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:10.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:11.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:11.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:11.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:11.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:12.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:12.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:12.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:12.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:13.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:13.174+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:13.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:13.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:13.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:13.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:14.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:14.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:14.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:14.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:15.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:15.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:15.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:15.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:16.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:16.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:16.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:16.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:17.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:17.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:17.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:17.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:18.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:18.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:18.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:18.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:18.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:18.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:19.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:19.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:19.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:19.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:20.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:20.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:20.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:20.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:21.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:21.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:21.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:21.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:22.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:22.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:22.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:22.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:23.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:23.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:23.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:23.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:23.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:23.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:24.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:24.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:24.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:24.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:25.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:25.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:25.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:25.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:26.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:26.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:27.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:27.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:27.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:27.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:28.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:28.177+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:28.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:28.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:28.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:28.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:29.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:29.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:29.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:29.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:30.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:30.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:30.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:30.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:31.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:31.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:31.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:31.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:32.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:32.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:32.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:32.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:33.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:33.177+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:33.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:33.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:33.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:33.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:34.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:34.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:34.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:34.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:35.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:35.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:35.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:35.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:37.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:37.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:37.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:37.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:37.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:37.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:37.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:37.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:38.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:38.177+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:38.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:38.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:38.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:38.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:40.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:40.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:40.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:40.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:40.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:40.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:40.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:40.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:41.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:41.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:41.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:41.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:42.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:42.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:42.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:42.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:43.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:43.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:43.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:43.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:43.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:43.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:44.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:44.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:44.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:44.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:45.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:45.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:45.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:46.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:46.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:46.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:46.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:47.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:47.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:43:14.206227+0000 (oldest deadline 2019-10-17T21:43:34.206227+0000) 2019-10-17T21:53:47.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:47.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:48.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:48.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:48.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:48.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:49.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:49.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:51.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:51.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:52.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:52.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:53.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:53.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:53.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:53.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:54.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:54.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:55.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:55.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:56.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:56.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:57.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:57.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:58.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:58.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:53:58.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:53:58.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:53:58.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:53:58.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:00.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:00.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:00.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:00.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:01.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:01.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:02.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:02.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:03.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:03.184+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:03.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:03.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:04.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:04.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:05.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:05.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:06.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:06.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:07.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:07.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:08.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:08.188+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:08.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:08.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:09.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:09.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:10.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:10.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:11.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:11.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:11.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:11.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:13.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:13.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:13.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:13.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:13.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:13.188+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:14.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:14.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:14.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:14.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:14.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:14.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:14.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:14.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:15.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:15.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:15.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:15.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:16.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:16.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:16.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:16.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:17.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:17.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:17.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:17.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:18.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:18.188+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:18.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:18.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:18.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:18.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:19.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:19.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:19.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:19.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:20.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:20.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:20.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:20.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:21.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:21.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:21.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:21.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:22.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:22.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:22.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:22.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:23.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:23.188+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:23.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:23.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:23.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:23.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:24.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:24.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:24.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:24.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:25.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:25.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:25.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:25.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:26.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:26.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:26.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:26.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:27.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:27.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:27.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:27.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:28.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:28.188+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:28.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:28.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:28.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:28.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:29.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:29.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:29.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:29.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:30.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:30.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:30.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:30.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:31.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:31.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:31.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:31.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:32.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:32.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:32.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:32.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:33.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:33.191+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:33.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:33.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:33.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:33.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:34.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:34.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:34.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:34.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:35.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:35.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:35.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:35.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:36.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:36.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:36.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:36.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:37.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:37.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:38.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:37.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:38.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:38.191+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:38.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:38.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:38.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:38.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:39.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:39.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:39.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:39.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:40.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:40.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:40.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:40.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:41.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:41.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:41.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:41.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:42.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:42.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:42.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:42.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:43.198 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:43.191+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:43.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:43.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:43.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:43.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:45.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:45.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:45.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:45.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:45.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:45.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:45.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:45.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:46.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:46.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:48.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:48.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:48.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:48.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:48.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:48.191+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:49.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:49.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:49.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:49.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:50.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:50.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:50.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:50.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:51.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:51.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:51.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:51.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:51.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:51.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:51.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:51.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:52.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:52.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:52.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:52.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:53.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:53.195+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:53.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:53.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:53.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:53.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:54.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:54.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:54.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:54.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:55.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:55.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:55.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:55.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:56.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:56.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:56.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:56.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:57.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:57.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:57.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:57.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:58.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:54:58.195+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:54:58.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:58.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:58.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:58.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:54:59.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:59.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:54:59.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:54:59.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:00.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:00.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:00.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:00.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:01.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:01.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:01.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:01.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:02.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:02.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:02.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:02.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:03.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:03.195+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:03.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:03.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:03.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:03.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:04.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:04.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:04.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:04.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:05.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:05.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:05.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:05.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:06.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:06.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:06.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:06.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:07.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:07.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:07.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:07.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:08.202 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:08.194+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:08.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:08.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:08.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:08.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:09.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:09.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:09.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:09.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:10.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:10.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:10.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:10.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:11.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:11.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:11.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:11.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:12.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:12.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:12.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:12.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:13.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:13.194+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:13.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:13.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:13.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:13.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:14.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:14.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:14.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:14.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:15.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:15.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:15.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:15.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:16.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:16.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:16.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:16.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:17.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:17.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:17.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:17.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:18.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:18.194+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:18.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:18.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:18.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:18.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:19.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:19.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:19.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:19.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:20.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:20.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:20.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:20.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:21.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:21.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:21.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:21.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:22.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:22.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:23.204 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:23.198+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:23.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:23.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:23.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:23.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:24.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:24.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:24.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:24.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:25.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:25.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:25.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:25.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:26.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:26.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:26.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:26.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:27.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:27.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:27.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:27.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:28.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:28.198+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:28.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:28.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:28.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:28.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:29.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:29.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:29.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:29.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:30.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:30.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:30.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:30.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:31.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:31.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:31.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:31.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:32.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:32.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:32.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:32.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:33.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:33.198+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:33.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:33.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:33.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:33.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:34.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:34.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:34.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:34.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:35.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:35.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:35.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:35.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:36.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:36.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:36.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:36.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:37.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:37.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:37.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:37.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:38.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:38.198+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:38.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:38.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:38.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:38.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:39.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:39.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:39.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:39.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:40.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:40.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:40.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:40.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:41.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:41.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:41.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:41.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:42.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:42.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:42.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:42.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:43.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:43.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:43.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:43.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:43.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:43.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:44.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:44.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:44.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:44.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:45.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:45.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:45.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:45.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:46.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:46.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:46.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:46.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:47.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:47.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:47.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:47.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:48.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:48.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:48.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:48.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:48.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:48.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:49.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:49.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:49.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:49.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:50.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:50.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:50.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:50.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:51.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:51.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:51.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:51.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:52.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:52.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:52.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:52.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:53.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:53.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:53.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:53.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:53.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:53.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:54.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:54.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:54.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:54.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:55.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:55.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:55.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:55.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:56.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:56.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:56.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:56.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:57.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:57.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:57.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:57.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:58.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:55:58.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:55:58.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:58.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:58.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:58.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:55:59.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:59.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:55:59.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:55:59.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:00.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:00.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:00.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:00.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:01.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:01.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:01.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:01.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:02.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:02.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:02.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:02.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:03.211 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:03.205+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:03.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:03.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:04.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:04.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:04.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:04.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:05.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:05.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:05.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:05.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:06.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:06.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:06.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:06.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:07.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:07.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:07.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:07.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:08.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:08.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:08.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:08.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:08.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:08.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:09.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:09.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:09.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:09.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:10.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:10.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:10.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:10.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:11.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:11.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:11.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:11.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:12.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:12.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:12.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:12.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:13.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:13.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:13.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:13.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:13.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:13.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:14.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:14.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:14.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:14.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:15.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:15.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:15.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:15.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:16.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:16.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:16.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:16.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:17.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:17.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:17.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:17.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:18.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:18.208+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:18.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:18.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:18.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:18.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:19.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:19.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:19.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:19.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:20.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:20.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:20.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:20.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:21.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:21.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:21.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:21.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:22.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:22.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:22.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:22.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:23.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:23.208+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:23.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:23.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:23.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:23.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:24.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:24.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:24.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:24.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:25.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:25.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:25.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:25.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:26.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:26.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:26.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:26.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:27.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:27.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:28.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:28.208+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:28.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:28.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:28.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:28.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:29.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:29.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:29.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:29.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:30.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:30.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:30.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:30.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:31.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:31.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:31.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:31.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:32.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:32.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:32.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:32.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:33.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:33.208+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:33.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:33.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:33.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:33.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:34.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:34.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:35.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:35.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:35.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:35.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:36.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:36.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:36.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:36.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:37.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:37.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:37.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:37.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:38.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:38.212+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:38.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:38.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:38.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:38.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:39.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:39.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:39.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:39.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:40.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:40.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:40.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:40.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:41.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:41.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:41.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:41.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:42.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:42.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:42.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:42.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:43.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:43.212+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:43.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:43.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:43.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:43.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:44.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:44.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:44.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:44.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:45.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:45.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:45.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:45.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:46.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:46.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:46.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:46.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:47.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:47.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:47.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:47.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:48.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:48.212+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:48.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:48.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:48.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:48.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:49.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:49.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:49.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:49.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:50.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:50.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:50.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:50.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:51.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:51.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:51.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:51.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:52.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:52.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:53.219 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:53.211+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:53.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:53.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:53.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:53.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:54.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:54.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:54.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:54.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:55.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:55.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:55.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:55.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:56.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:56.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:56.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:56.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:57.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:57.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:57.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:57.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:58.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:56:58.215+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:56:58.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:58.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:58.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:58.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:56:59.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:59.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:56:59.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:56:59.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:00.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:00.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:00.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:00.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:01.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:01.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:01.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:01.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:02.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:02.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:02.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:02.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:03.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:03.215+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:03.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:03.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:03.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:03.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:04.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:04.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:04.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:04.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:05.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:05.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:05.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:05.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:06.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:06.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:06.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:06.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:07.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:07.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:07.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:07.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:08.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:08.215+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:08.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:08.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:08.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:08.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:09.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:09.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:09.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:09.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:10.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:10.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:10.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:10.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:11.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:11.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:11.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:11.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:13.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:13.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:13.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:13.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:13.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:13.215+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:14.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:14.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:14.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:14.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:15.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:15.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:15.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:15.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:16.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:16.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:16.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:16.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:16.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:16.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:16.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:16.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:18.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:17.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:18.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:17.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:18.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:18.215+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:18.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:18.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:18.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:18.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:20.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:20.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:20.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:20.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:21.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:21.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:21.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:21.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:22.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:22.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:22.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:22.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:23.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:23.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:23.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:23.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:23.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:23.215+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:24.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:24.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:24.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:24.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:25.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:25.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:25.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:25.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:26.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:26.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:26.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:26.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:27.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:27.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:27.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:27.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:28.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:28.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:28.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:28.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:28.225 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:28.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:29.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:29.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:29.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:29.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:30.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:30.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:30.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:30.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:31.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:31.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:31.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:31.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:32.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:32.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:32.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:32.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:33.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:33.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:33.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:33.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:33.225 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:33.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:34.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:34.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:34.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:34.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:35.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:35.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:35.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:35.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:36.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:36.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:36.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:36.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:37.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:37.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:37.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:37.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:38.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:38.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:38.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:38.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:38.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:38.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:39.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:39.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:39.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:39.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:40.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:40.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:40.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:40.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:41.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:41.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:42.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:42.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:42.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:42.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:43.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:43.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:43.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:43.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:43.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:43.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:44.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:44.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:44.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:44.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:45.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:45.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:45.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:45.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:46.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:46.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:46.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:46.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:47.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:47.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:47.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:47.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:48.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:48.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:48.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:48.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:48.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:48.222+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:49.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:49.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:49.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:49.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:50.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:50.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:50.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:50.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:51.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:51.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:51.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:51.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:52.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:52.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:52.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:52.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:53.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:53.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:53.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:53.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:53.222+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:54.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:54.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:54.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:54.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:55.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:55.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:55.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:55.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:56.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:56.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:56.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:56.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:57.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:57.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:57.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:57.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:58.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:58.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:58.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:58.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:57:58.229 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:57:58.222+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:57:59.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:59.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:57:59.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:57:59.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:00.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:00.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:00.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:00.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:01.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:01.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:01.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:01.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:02.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:02.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:02.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:02.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:02.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:02.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:02.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:02.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:03.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:03.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:03.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:03.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:03.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:03.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:04.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:04.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:04.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:04.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:05.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:05.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:05.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:05.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:06.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:06.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:06.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:06.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:07.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:07.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:07.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:07.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:08.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:08.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:08.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:08.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:08.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:08.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:09.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:09.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:09.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:09.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:10.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:10.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:10.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:10.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:11.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:11.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:11.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:11.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:12.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:12.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:12.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:12.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:13.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:13.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:13.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:13.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:13.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:13.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:14.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:14.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:14.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:14.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:15.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:15.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:15.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:15.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:17.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:17.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:17.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:17.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 48 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:17.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:17.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:17.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:17.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:18.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:18.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:18.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:18.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:18.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:18.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:19.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:19.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:19.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:19.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:20.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:20.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:20.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:20.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:21.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:21.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:21.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:21.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:22.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:22.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:22.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:22.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:23.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:23.229+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:23.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:23.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:23.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:23.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:24.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:24.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:24.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:24.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:25.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:25.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:25.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:25.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:26.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:26.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:26.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:26.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:27.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:27.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:27.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:27.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:28.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:28.229+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:28.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:28.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:28.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:28.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:29.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:29.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:29.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:29.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:30.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:30.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:31.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:31.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:31.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:31.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:32.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:32.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:32.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:32.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:33.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:33.229+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:33.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:33.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:33.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:33.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:34.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:34.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:34.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:34.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:35.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:35.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:35.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:35.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:36.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:36.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:36.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:36.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:37.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:37.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:37.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:37.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:38.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:38.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:38.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:38.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:38.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:38.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:39.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:39.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:39.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:39.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:40.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:40.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:40.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:40.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:41.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:41.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:41.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:41.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:42.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:42.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:42.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:42.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:43.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:43.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:43.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:43.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:43.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:43.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:44.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:44.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:44.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:44.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:45.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:45.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:45.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:45.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:46.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:46.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:46.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:46.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:47.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:47.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:47.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:47.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:48.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:48.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:48.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:48.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:48.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:48.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:49.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:49.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:49.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:49.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:50.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:50.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:50.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:50.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:51.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:51.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:51.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:51.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:52.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:52.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:52.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:52.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:53.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:53.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:53.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:53.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:53.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:53.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:54.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:54.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:54.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:54.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:55.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:55.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:55.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:55.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:56.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:56.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:56.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:56.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:57.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:57.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:57.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:57.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:58.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:58:58.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:58:58.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:58.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:58.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:58.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:58:59.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:59.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:58:59.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:58:59.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:00.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:00.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:00.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:00.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:01.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:01.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:01.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:01.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:02.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:02.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:02.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:02.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:03.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:03.236+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:04.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:04.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:04.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:04.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:05.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:05.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:05.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:05.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:05.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:05.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:06.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:06.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:06.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:06.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:07.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:07.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:07.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:07.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:08.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:08.236+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:08.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:08.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:08.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:08.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:09.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:09.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:09.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:09.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:10.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:10.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:10.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:10.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:11.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:11.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:11.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:11.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:12.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:12.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:12.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:12.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:13.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:13.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:13.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:13.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:13.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:13.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:14.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:14.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:14.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:14.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:15.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:15.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:15.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:15.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:16.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:16.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:16.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:16.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:18.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:17.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:18.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:17.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:18.242 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:18.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:19.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:19.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:19.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:19.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:20.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:20.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:20.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:20.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:20.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:20.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:20.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:20.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:21.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:21.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:21.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:21.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:22.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:22.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:22.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:22.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:23.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:23.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:23.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:23.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:23.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:23.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:24.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:24.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:24.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:24.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:25.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:25.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:25.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:25.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:26.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:26.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:26.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:26.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:27.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:27.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:27.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:27.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:28.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:28.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:28.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:28.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:28.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:28.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:29.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:29.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:29.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:29.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:30.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:30.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:30.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:30.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:31.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:31.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:31.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:31.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:32.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:32.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:32.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:33.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:33.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:33.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:33.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:33.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:33.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:34.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:34.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:34.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:34.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:35.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:35.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:35.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:35.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:37.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:36.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:37.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:36.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:37.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:37.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:37.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:37.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:38.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:38.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:38.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:38.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:38.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:38.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:39.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:39.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:39.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:39.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:40.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:40.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:40.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:40.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:42.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:42.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:42.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:42.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:43.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:42.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:43.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:42.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:43.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:43.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:43.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:43.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:43.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:43.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:44.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:44.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:44.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:44.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:45.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:45.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:45.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:45.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:46.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:46.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:46.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:46.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:47.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:47.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:47.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:47.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:48.247 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:48.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:48.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:48.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:48.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:48.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:49.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:49.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:49.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:49.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:50.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:50.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:50.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:51.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:51.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:51.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:51.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:52.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:52.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:52.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:52.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:53.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:53.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:53.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:53.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:53.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:53.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:54.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:54.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:54.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:54.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:55.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:55.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:55.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:55.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:56.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:56.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:56.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:56.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:57.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:57.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:57.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:57.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:58.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T21:59:58.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T21:59:58.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:58.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:58.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:58.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T21:59:59.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:59.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T21:59:59.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T21:59:59.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:00.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:00.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:00.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:00.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:01.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:01.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:01.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:01.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:02.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:02.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:02.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:02.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:03.250 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:03.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:03.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:03.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:03.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:03.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:04.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:04.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:04.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:04.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:05.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:05.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:05.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:05.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:06.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:06.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:06.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:06.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:07.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:07.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:07.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:07.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:08.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:08.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:08.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:08.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:08.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:08.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:09.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:09.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:09.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:09.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:10.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:10.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:10.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:10.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:11.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:11.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:11.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:11.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:12.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:12.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:12.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:12.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:13.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:13.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:13.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:13.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:13.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:13.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:14.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:14.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:14.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:14.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:15.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:15.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:15.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:15.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:16.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:16.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:16.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:16.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:17.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:17.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:17.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:17.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:18.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:18.246+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:18.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:18.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:18.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:18.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:19.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:19.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:19.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:19.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:20.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:20.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:20.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:20.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:21.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:21.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:21.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:21.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:22.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:22.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:22.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:22.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:23.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:23.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:23.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:23.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:23.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:23.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:24.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:24.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:25.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:25.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:25.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:25.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:26.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:26.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:26.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:26.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:27.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:27.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:27.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:27.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:28.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:28.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:28.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:28.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:28.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:28.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:29.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:29.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:29.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:29.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:30.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:30.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:30.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:30.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:31.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:31.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:31.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:31.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:32.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:32.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:32.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:32.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:33.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:33.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:33.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:33.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:33.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:33.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:34.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:34.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:34.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:34.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:35.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:35.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:35.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:35.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:36.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:36.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:36.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:36.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:37.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:37.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:37.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:37.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:38.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:38.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:38.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:38.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:38.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:38.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:39.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:39.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:39.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:39.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:40.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:40.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:40.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:40.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:41.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:41.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:41.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:41.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:42.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:42.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:42.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:42.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:43.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:43.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:43.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:43.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:43.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:43.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:44.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:44.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:44.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:44.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:45.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:45.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:45.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:45.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:46.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:46.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:46.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:46.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:47.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:47.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:47.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:47.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:48.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:48.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:48.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:48.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:48.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:48.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:49.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:49.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:49.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:49.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:50.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:50.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:50.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:50.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:51.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:51.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:51.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:51.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:52.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:52.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:53.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:53.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:53.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:53.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:53.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:53.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:54.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:54.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:54.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:54.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:55.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:55.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:55.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:55.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:56.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:56.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:56.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:56.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:57.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:57.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:57.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:57.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:58.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:00:58.252+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:00:58.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:58.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:58.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:58.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:00:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:59.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:00:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:00:59.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:00.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:00.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:00.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:00.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:01.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:01.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:01.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:01.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:02.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:02.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:02.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:02.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:03.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:03.252+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:03.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:03.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:03.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:03.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:04.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:04.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:04.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:04.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:05.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:05.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:05.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:05.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:06.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:06.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:06.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:06.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:07.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:07.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:07.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:08.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:08.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:08.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:08.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:08.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:08.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:09.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:09.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:09.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:09.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:10.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:10.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:10.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:10.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:11.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:11.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:11.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:11.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:12.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:12.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:12.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:12.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:13.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:13.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:13.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:13.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:13.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:13.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:14.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:14.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:15.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:15.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:15.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:15.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:16.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:16.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:16.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:16.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:17.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:17.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:17.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:17.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:18.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:18.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:18.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:18.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:18.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:18.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:19.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:19.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:19.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:19.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:20.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:20.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:20.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:20.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:21.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:21.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:21.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:21.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:22.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:22.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:22.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:22.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:23.263 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:23.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:23.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:23.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:23.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:23.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:24.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:24.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:24.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:24.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:25.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:25.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:25.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:25.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:26.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:26.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:26.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:26.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:27.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:27.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:27.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:27.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:28.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:28.260+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:28.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:28.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:28.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:28.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:29.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:29.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:29.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:29.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:30.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:30.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:30.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:30.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:31.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:31.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:31.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:31.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:32.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:32.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:32.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:32.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:33.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:33.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:33.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:33.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:33.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:33.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:34.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:34.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:34.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:34.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:35.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:35.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:35.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:35.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:36.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:36.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:36.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:36.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:37.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:37.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:37.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:37.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:38.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:38.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:38.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:38.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:38.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:38.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:39.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:39.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:39.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:39.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:40.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:40.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:40.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:40.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:41.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:41.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:41.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:41.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:42.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:42.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:42.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:42.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:43.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:43.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:43.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:43.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:43.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:43.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:44.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:44.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:44.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:44.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:45.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:45.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:45.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:45.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:46.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:46.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:46.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:46.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:47.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:47.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:47.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:47.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:48.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:48.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:48.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:48.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:48.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:48.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:49.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:49.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:49.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:49.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:50.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:50.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:50.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:50.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:51.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:51.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:51.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:51.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:52.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:52.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:52.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:52.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:53.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:53.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:53.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:53.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:53.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:53.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:54.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:54.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:54.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:54.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:56.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:55.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:56.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:55.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:57.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:56.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:57.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:56.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:57.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:57.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:57.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:57.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:58.269 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:01:58.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:01:58.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:58.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:58.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:58.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:01:59.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:59.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:01:59.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:01:59.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:01.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:01.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:01.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:01.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:02.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:02.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:02.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:02.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:02.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:02.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:02.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:02.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:03.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:03.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:03.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:03.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:03.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:03.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:05.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:05.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:05.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:05.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:06.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:06.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:06.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:06.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:06.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:06.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:06.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:06.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:07.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:07.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:07.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:07.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:08.271 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:08.262+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:09.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:08.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:09.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:08.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:09.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:09.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:09.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:09.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:10.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:10.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:10.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:10.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:11.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:11.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:11.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:11.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:12.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:12.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:12.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:12.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:13.271 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:13.262+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:13.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:13.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:13.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:13.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:15.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:15.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:15.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:15.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:16.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:16.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:16.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:16.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:17.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:17.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:17.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:17.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:18.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:17.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:18.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:17.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:18.272 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:18.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:19.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:19.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:19.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:19.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:20.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:20.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:20.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:20.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:20.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:20.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:20.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:20.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:21.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:21.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:21.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:21.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:22.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:22.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:22.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:22.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:23.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:23.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:23.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:23.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:23.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:23.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:24.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:24.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:24.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:24.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:25.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:25.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:25.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:25.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:26.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:26.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:26.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:26.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:27.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:27.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:27.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:27.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:28.274 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:28.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:28.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:28.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:28.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:28.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:29.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:29.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:29.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:29.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:30.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:30.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:30.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:30.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:31.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:31.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:31.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:31.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:32.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:32.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:32.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:32.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:33.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:33.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:33.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:33.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:33.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:33.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:34.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:34.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:34.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:34.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:35.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:35.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:35.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:35.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:36.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:36.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:36.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:36.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:37.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:37.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:37.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:37.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:38.276 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:38.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:38.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:38.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:38.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:38.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:39.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:39.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:39.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:39.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:40.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:40.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:40.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:40.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:41.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:41.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:41.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:41.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:42.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:42.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:42.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:42.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:43.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:43.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:43.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:43.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:43.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:43.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:44.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:44.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:44.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:44.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:45.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:45.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:45.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:45.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:47.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:46.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:47.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:46.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:48.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:48.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:48.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:48.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:48.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:48.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:48.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:48.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:48.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:48.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:50.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:50.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:50.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:50.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:51.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:51.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:51.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:51.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:52.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:52.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:52.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:52.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:53.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:52.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:53.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:52.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:53.278 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:53.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:54.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:54.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:54.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:54.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:54.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:54.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:54.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:54.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:56.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:56.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:56.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:56.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:57.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:57.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:57.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:57.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:58.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:58.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:58.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:58.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:02:58.278 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:02:58.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:02:59.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:59.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:02:59.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:02:59.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:00.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:00.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:00.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:00.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:01.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:01.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:01.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:01.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:02.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:02.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:03.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:03.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:03.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:03.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:03.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:03.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:04.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:04.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:04.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:04.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:05.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:05.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:05.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:05.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:06.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:06.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:06.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:06.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:07.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:07.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:07.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:07.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:08.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:08.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:08.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:08.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:08.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:08.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:09.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:09.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:09.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:09.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:10.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:10.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:10.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:10.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:11.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:11.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:11.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:11.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:12.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:12.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:12.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:12.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:13.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:13.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:13.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:13.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:13.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:13.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:14.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:14.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:14.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:14.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:15.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:15.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:15.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:15.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:16.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:16.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:16.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:16.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:17.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:17.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:17.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:17.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:18.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:18.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:18.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:18.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:18.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:18.276+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:19.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:19.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:19.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:19.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:20.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:20.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:20.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:20.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:21.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:21.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:21.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:21.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:22.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:22.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:22.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:22.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:23.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:23.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:23.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:23.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:23.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:23.276+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:24.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:24.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:24.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:24.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:25.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:25.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:26.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:26.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:26.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:26.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:27.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:27.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:27.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:27.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:28.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:28.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:28.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:28.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:28.284 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:28.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:29.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:29.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:29.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:29.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:30.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:30.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:30.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:30.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:31.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:31.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:31.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:31.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:32.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:32.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:32.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:32.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:33.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:33.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:33.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:33.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:33.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:33.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:34.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:34.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:34.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:34.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:35.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:35.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:35.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:35.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:36.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:36.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:36.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:36.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:37.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:37.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:37.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:37.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:38.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:38.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:38.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:38.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:38.286 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:38.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:39.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:39.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:39.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:39.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:40.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:40.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:40.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:40.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:41.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:41.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:41.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:41.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:42.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:42.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:42.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:42.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:43.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:43.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:43.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:43.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:43.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:43.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:44.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:44.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:44.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:44.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:45.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:45.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:45.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:45.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:46.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:46.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:46.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:46.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:47.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:47.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:47.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:47.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:48.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:48.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:48.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:48.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:48.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:48.284+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:49.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:49.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:49.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:49.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:50.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:50.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:50.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:50.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:51.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:51.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:51.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:51.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:52.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:52.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:52.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:52.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:53.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:53.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:53.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:53.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:53.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:53.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:54.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:54.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:54.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:54.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:55.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:55.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:55.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:55.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:56.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:56.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:56.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:56.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:57.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:57.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:57.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:57.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:58.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:58.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:58.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:58.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:03:58.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:03:58.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:03:59.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:59.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:03:59.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:03:59.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:00.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:00.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:00.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:00.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:01.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:01.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:01.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:01.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:02.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:02.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:02.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:02.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:03.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:03.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:03.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:03.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:03.290 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:03.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:04.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:04.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:04.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:04.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:05.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:05.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:05.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:05.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:06.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:06.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:06.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:06.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:07.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:07.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:07.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:07.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:08.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:08.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:08.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:08.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:08.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:08.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:09.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:09.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:09.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:09.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:10.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:10.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:10.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:10.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:11.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:11.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:11.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:11.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:12.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:12.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:12.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:12.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:13.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:13.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:13.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:13.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:13.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:13.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:14.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:14.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:14.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:14.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:15.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:15.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:15.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:15.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:16.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:16.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:16.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:16.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:17.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:17.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:17.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:17.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:18.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:18.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:18.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:18.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:18.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:18.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:19.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:19.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:19.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:19.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:20.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:20.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:20.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:20.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:21.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:21.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:21.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:21.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:22.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:22.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:22.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:22.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:23.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:23.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T21:53:51.333287+0000 (oldest deadline 2019-10-17T21:54:11.333287+0000) 2019-10-17T22:04:23.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:23.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:23.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:23.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:24.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:24.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:25.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:25.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:26.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:26.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:27.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:27.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:28.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:28.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:28.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:28.286+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:29.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:29.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:30.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:30.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:31.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:31.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:32.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:32.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:33.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:33.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:33.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:33.286+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:34.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:34.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:35.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:35.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:36.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:36.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:37.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:37.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:38.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:38.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:38.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:38.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:39.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:39.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:40.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:40.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:41.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:41.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:42.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:42.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:43.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:43.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:43.297 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:43.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:44.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:44.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:45.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:45.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:46.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:46.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:47.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:47.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:47.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:47.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:48.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:48.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:48.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:48.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:48.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:48.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:49.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:49.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:49.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:49.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:50.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:50.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:50.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:50.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:51.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:51.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:51.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:51.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:52.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:52.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:52.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:52.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:53.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:53.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:53.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:53.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:53.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:53.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:54.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:54.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:54.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:54.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:55.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:55.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:55.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:55.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:56.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:56.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:56.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:56.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:57.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:57.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:57.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:57.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:58.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:58.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:58.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:58.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:04:58.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:04:58.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:04:59.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:59.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:04:59.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:04:59.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:00.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:00.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:00.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:00.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:01.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:01.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:01.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:01.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:02.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:02.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:02.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:02.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:03.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:03.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:03.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:03.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:03.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:03.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:04.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:04.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:04.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:04.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:05.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:05.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:05.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:05.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:06.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:06.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:06.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:06.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:07.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:07.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:07.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:07.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:08.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:08.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:08.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:08.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:08.301 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:08.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:09.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:09.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:09.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:09.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:10.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:10.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:10.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:10.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:11.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:11.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:11.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:11.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:12.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:12.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:12.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:12.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:13.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:13.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:13.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:13.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:13.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:13.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:14.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:14.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:14.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:14.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:15.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:15.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:15.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:15.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:16.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:16.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:17.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:17.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:17.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:17.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:18.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:18.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:18.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:18.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:18.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:18.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:19.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:19.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:19.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:19.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:20.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:20.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:20.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:20.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:21.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:21.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:21.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:21.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:22.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:22.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:22.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:22.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:23.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:23.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:23.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:23.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:23.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:23.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:24.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:24.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:24.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:24.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:25.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:25.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:25.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:25.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:26.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:26.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:26.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:26.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:27.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:27.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:27.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:27.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:28.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:28.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:28.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:28.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:28.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:28.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:29.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:29.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:29.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:29.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:30.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:30.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:30.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:30.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:31.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:31.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:31.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:31.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:32.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:32.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:32.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:32.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:33.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:33.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:33.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:33.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:33.306 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:33.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:33.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:33.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:33.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:33.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:34.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:34.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:34.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:34.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:35.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:35.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:35.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:35.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:36.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:36.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:36.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:36.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:37.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:37.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:37.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:37.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:38.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:38.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:38.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:38.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:38.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:38.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:39.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:39.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:39.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:39.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:40.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:40.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:40.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:40.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:42.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:42.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:42.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:42.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:43.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:43.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:43.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:43.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:43.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:43.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:44.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:44.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:44.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:44.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:45.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:45.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:45.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:45.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:45.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:45.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:45.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:45.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:46.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:46.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:46.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:46.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:47.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:47.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:47.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:47.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:48.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:48.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:48.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:48.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:48.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:48.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:49.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:49.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:49.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:49.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:50.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:50.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:50.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:50.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:51.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:51.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:51.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:51.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:52.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:52.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:52.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:52.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:53.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:53.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:53.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:53.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:53.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:53.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:54.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:54.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:54.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:54.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:55.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:55.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:55.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:55.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:56.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:56.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:56.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:56.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:57.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:57.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:57.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:57.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:58.310 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:05:58.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:05:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:58.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:58.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:05:59.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:59.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:05:59.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:05:59.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:00.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:00.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:00.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:00.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:01.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:01.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:01.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:01.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:02.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:02.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:02.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:02.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:03.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:03.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:03.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:03.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:03.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:03.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:04.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:04.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:04.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:04.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:05.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:05.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:05.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:05.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:06.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:06.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:06.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:06.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:07.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:07.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:07.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:07.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:08.312 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:08.308+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:08.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:08.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:08.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:08.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:09.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:09.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:09.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:09.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:10.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:10.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:10.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:10.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:11.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:11.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:11.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:11.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:12.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:12.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:12.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:12.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:13.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:13.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:13.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:13.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:13.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:13.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:14.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:14.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:14.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:15.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:15.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:15.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:15.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:16.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:16.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:16.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:16.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:17.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:17.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:17.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:17.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:18.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:18.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:18.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:18.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:18.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:18.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:19.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:19.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:19.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:19.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:20.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:20.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:20.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:20.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:21.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:21.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:21.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:21.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:22.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:22.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:22.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:22.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:23.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:23.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:23.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:23.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:23.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:23.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:24.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:24.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:24.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:24.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:26.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:26.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:26.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:26.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:26.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:26.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:26.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:26.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:28.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:28.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:28.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:28.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:28.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:28.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:28.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:28.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:28.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:28.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:29.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:29.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:29.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:29.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:30.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:30.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:30.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:30.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:31.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:31.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:31.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:31.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:32.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:32.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:32.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:32.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:33.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:33.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:33.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:33.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:33.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:33.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:34.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:34.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:34.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:34.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:35.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:35.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:35.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:35.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:36.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:36.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:36.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:36.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:37.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:37.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:37.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:37.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:38.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:38.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:38.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:38.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:38.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:38.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:39.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:39.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:39.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:39.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:40.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:40.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:40.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:40.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:41.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:41.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:41.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:41.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:42.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:42.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:42.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:42.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:43.318 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:43.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:43.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:43.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:43.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:43.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:44.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:44.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:44.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:44.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:45.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:45.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:45.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:45.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:46.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:46.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:46.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:46.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:47.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:47.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:47.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:47.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:48.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:48.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:48.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:48.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:48.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:48.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:49.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:49.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:49.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:49.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:50.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:50.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:50.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:50.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:51.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:51.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:51.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:51.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:52.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:52.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:52.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:52.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:53.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:53.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:53.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:53.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:53.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:53.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:54.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:54.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:54.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:54.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:55.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:55.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:55.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:55.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:56.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:56.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:56.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:56.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:57.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:57.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:57.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:57.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:58.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:06:58.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:06:58.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:58.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:58.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:58.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:06:59.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:59.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:06:59.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:06:59.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:00.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:00.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:00.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:00.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:01.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:01.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:01.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:01.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:02.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:02.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:02.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:02.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:03.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:03.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:03.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:03.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:03.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:03.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:04.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:04.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:05.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:05.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:05.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:05.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:06.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:06.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:06.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:06.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:07.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:07.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:07.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:07.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:08.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:08.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:08.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:08.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:08.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:08.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:09.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:09.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:09.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:09.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:10.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:10.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:10.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:11.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:11.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:11.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:11.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:12.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:12.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:12.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:12.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:13.323 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:13.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:13.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:13.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:13.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:13.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:14.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:14.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:14.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:14.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:15.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:15.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:15.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:15.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:16.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:16.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:16.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:16.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:17.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:17.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:17.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:17.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:18.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:18.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:18.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:18.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:18.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:18.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:19.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:19.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:19.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:19.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:20.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:20.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:20.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:20.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:21.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:21.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:21.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:21.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:22.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:22.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:22.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:22.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:23.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:23.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:23.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:23.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:23.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:23.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:24.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:24.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:24.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:24.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:25.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:25.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:25.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:25.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:26.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:26.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:26.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:26.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:27.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:27.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:27.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:27.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:28.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:28.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:28.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:28.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:28.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:28.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:29.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:29.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:29.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:29.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:30.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:30.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:30.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:30.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:31.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:31.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:31.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:31.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:32.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:32.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:32.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:32.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:33.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:33.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:33.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:33.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:33.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:33.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:34.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:34.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:34.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:34.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:35.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:35.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:35.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:35.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:36.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:36.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:36.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:36.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:37.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:37.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:37.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:37.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:38.327 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:38.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:38.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:38.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:38.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:38.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:39.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:39.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:39.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:39.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:40.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:40.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:40.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:40.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:41.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:41.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:41.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:41.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:42.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:42.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:42.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:42.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:43.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:43.325+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:43.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:43.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:43.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:43.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:44.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:44.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:44.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:44.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:45.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:45.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:45.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:45.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:46.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:46.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:46.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:46.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:47.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:47.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:47.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:47.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:48.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:48.325+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:48.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:48.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:48.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:48.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:49.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:49.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:49.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:49.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:50.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:50.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:50.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:50.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:51.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:51.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:51.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:51.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:52.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:52.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:52.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:52.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:53.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:53.325+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:53.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:53.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:53.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:53.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:54.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:54.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:54.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:54.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:55.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:55.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:55.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:55.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:56.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:56.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:56.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:56.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:57.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:57.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:57.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:57.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:58.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:07:58.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:07:58.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:58.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:58.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:58.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:07:59.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:59.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:07:59.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:07:59.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:00.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:00.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:00.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:00.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:01.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:01.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:01.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:02.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:02.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:02.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:02.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:03.331 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:03.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:03.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:03.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:03.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:03.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:04.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:04.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:04.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:04.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:05.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:05.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:05.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:05.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:06.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:06.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:07.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:07.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:07.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:07.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:08.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:08.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:08.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:08.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:08.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:08.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:09.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:09.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:09.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:09.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:10.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:10.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:10.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:10.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:11.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:11.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:11.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:11.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:12.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:12.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:12.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:12.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:13.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:13.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:13.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:13.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:13.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:13.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:14.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:14.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:14.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:14.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:15.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:15.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:15.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:15.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:16.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:16.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:16.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:16.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:17.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:17.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:17.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:17.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:18.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:18.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:18.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:18.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:18.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:18.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:19.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:19.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:19.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:19.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:20.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:20.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:20.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:20.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:21.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:21.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:21.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:21.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:22.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:22.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:22.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:22.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:23.335 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:23.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:23.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:23.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:23.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:23.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:24.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:24.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:24.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:24.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:25.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:25.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:25.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:25.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:26.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:26.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:26.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:26.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:27.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:27.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:27.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:27.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:28.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:28.332+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:28.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:28.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:28.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:28.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:29.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:29.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:29.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:29.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:30.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:30.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:30.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:30.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:31.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:31.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:31.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:31.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:32.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:32.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:32.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:32.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:33.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:33.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:33.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:33.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:33.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:33.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:34.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:34.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:34.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:34.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:35.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:35.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:35.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:35.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:36.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:36.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:36.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:36.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:37.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:37.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:37.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:37.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:38.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:38.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:38.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:38.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:38.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:38.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:39.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:39.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:39.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:39.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:40.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:40.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:40.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:40.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:41.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:41.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:41.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:41.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:42.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:42.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:42.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:42.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:43.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:43.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:43.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:43.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:43.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:43.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:44.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:44.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:44.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:44.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:45.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:45.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:45.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:45.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:46.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:46.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:46.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:46.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:47.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:47.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:47.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:47.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:48.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:48.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:48.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:48.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:49.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:49.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:49.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:49.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:50.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:50.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:50.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:50.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:51.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:51.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:51.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:51.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:52.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:52.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:53.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:53.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:53.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:53.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:53.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:53.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:54.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:54.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:54.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:54.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:55.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:55.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:55.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:55.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:56.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:56.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:56.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:56.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:57.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:57.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:57.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:57.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:58.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:08:58.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:08:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:58.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:58.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:08:59.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:59.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:08:59.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:08:59.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:00.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:00.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:01.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:01.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:01.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:01.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:02.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:02.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:02.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:02.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:03.342 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:03.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:03.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:03.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:03.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:03.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:04.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:04.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:04.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:04.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:05.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:05.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:05.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:05.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:06.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:06.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:06.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:06.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:07.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:07.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:07.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:07.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:08.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:08.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:08.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:08.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:08.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:08.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:09.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:09.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:09.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:09.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:10.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:10.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:10.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:10.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:11.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:11.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:11.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:11.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:12.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:12.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:12.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:12.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:13.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:13.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:13.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:13.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:13.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:13.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:14.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:14.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:14.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:14.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:15.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:15.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:15.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:15.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:16.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:16.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:16.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:16.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:17.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:17.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:17.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:17.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:18.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:18.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:18.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:18.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:18.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:18.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:19.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:19.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:19.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:19.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:20.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:20.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:20.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:20.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:21.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:21.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:21.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:21.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:22.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:22.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:22.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:22.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:23.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:23.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:23.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:23.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:23.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:23.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:24.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:24.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:24.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:24.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:25.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:25.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:25.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:25.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:26.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:26.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:26.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:26.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:27.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:27.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:27.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:27.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:28.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:28.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:28.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:28.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:28.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:28.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:29.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:29.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:29.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:29.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:30.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:30.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:30.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:30.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:31.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:31.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:31.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:31.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:32.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:32.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:32.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:32.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:33.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:33.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:33.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:33.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:33.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:33.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:34.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:34.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:34.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:34.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:35.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:35.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:35.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:35.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:36.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:36.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:36.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:36.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:37.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:37.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:37.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:37.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:38.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:38.342+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:38.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:38.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:38.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:39.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:39.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:40.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:40.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:40.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:40.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:41.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:41.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:41.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:41.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:42.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:42.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:42.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:42.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:43.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:43.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:43.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:43.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:43.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:43.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:44.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:44.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:44.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:44.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:45.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:45.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:45.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:45.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:46.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:46.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:46.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:46.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:47.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:47.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:47.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:47.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:48.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:48.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:48.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:48.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:48.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:48.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:49.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:49.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:49.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:49.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:50.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:50.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:50.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:50.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:51.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:51.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:51.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:51.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:52.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:52.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:52.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:52.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:53.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:53.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:53.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:53.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:53.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:53.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:54.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:54.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:54.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:54.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:55.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:55.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:55.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:55.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:56.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:56.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:56.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:56.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:57.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:57.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:57.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:57.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:58.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:09:58.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:09:58.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:58.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:58.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:58.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:09:59.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:59.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:09:59.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:09:59.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:00.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:00.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:00.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:00.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:01.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:01.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:01.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:01.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:02.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:02.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:02.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:02.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:03.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:03.349+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:03.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:03.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:03.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:03.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:04.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:04.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:04.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:04.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:05.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:05.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:05.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:05.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:06.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:06.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:06.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:06.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:07.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:07.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:07.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:07.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:08.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:08.349+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:08.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:08.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:08.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:08.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:09.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:09.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:09.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:09.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:10.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:10.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:10.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:10.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:11.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:11.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:11.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:11.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:12.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:12.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:12.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:12.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:13.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:13.349+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:13.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:13.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:13.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:13.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:14.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:14.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:14.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:14.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:15.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:15.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:15.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:15.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:16.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:16.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:16.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:16.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:17.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:17.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:17.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:17.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:18.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:18.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:18.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:18.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:18.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:18.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:19.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:19.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:19.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:19.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:20.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:20.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:20.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:20.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:21.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:21.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:21.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:21.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:22.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:22.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:22.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:22.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:23.356 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:23.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:23.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:23.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:24.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:24.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:24.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:24.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:25.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:25.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:25.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:25.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:26.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:26.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:26.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:26.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:27.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:27.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:27.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:27.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:28.356 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:28.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:28.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:28.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:28.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:28.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:29.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:29.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:29.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:29.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:30.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:30.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:30.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:30.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:31.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:31.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:31.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:31.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:32.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:32.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:32.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:32.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:33.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:33.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:33.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:33.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:33.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:33.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:34.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:34.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:34.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:34.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:35.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:35.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:35.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:35.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:36.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:36.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:36.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:36.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:37.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:37.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:37.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:37.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:38.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:38.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:38.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:38.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:38.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:38.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:39.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:39.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:39.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:39.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:40.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:40.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:40.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:40.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:41.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:41.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:41.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:41.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:42.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:42.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:42.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:42.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:43.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:43.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:43.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:43.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:43.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:43.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:44.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:44.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:44.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:44.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:45.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:45.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:45.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:45.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:46.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:46.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:46.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:46.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:47.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:47.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:47.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:47.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:48.360 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:48.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:48.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:48.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:48.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:48.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:49.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:49.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:49.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:49.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:50.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:50.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:50.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:50.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:51.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:51.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:51.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:51.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:52.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:52.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:52.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:52.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:53.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:53.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:53.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:53.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:53.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:53.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:54.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:54.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:54.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:54.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:55.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:55.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:55.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:55.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:56.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:56.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:56.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:56.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:57.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:57.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:57.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:57.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:58.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:10:58.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:10:58.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:58.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:58.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:58.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:10:59.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:59.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:10:59.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:10:59.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:00.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:00.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:00.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:00.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:01.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:01.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:01.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:01.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:02.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:02.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:02.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:02.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:03.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:03.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:03.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:03.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:03.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:03.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:04.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:04.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:04.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:04.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:05.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:05.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:05.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:05.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:06.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:06.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:06.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:06.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:07.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:07.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:07.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:07.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:08.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:08.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:08.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:08.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:08.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:08.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:09.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:09.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:09.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:09.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:10.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:10.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:10.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:10.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:11.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:11.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:11.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:11.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:12.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:12.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:12.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:12.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:13.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:13.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:13.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:13.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:13.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:13.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:14.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:14.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:14.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:14.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:15.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:15.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:15.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:15.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:16.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:16.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:16.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:16.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:17.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:17.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:17.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:17.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:18.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:18.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:18.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:18.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:18.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:18.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:19.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:19.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:19.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:19.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:20.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:20.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:20.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:20.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:21.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:21.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:21.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:21.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:22.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:22.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:22.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:22.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:23.366 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:23.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:23.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:23.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:23.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:23.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:24.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:24.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:24.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:24.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:25.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:25.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:25.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:25.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:26.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:26.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:26.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:26.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:27.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:27.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:27.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:27.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:28.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:28.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:28.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:28.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:28.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:28.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:29.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:29.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:29.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:29.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:30.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:30.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:30.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:30.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:31.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:31.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:31.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:31.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:32.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:32.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:32.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:32.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:33.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:33.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:33.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:33.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:33.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:33.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:34.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:34.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:34.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:34.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:35.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:35.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:35.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:35.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:36.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:36.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:36.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:36.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:37.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:37.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:37.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:37.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:38.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:38.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:38.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:38.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:38.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:38.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:39.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:39.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:39.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:39.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:40.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:40.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:40.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:40.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:41.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:41.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:41.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:41.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:42.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:42.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:42.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:42.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:43.369 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:43.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:43.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:43.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:43.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:43.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:44.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:44.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:44.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:44.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:45.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:45.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:45.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:45.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:46.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:46.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:46.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:46.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:47.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:47.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:47.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:47.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:48.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:48.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:48.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:48.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:48.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:48.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:49.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:49.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:49.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:49.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:50.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:50.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:50.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:50.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:51.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:51.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:52.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:52.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:52.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:52.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:53.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:53.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:53.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:53.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:53.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:53.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:54.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:54.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:54.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:54.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:55.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:55.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:55.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:55.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:56.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:56.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:56.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:56.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:57.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:57.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:57.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:57.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:58.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:11:58.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:11:58.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:58.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:58.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:58.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:11:59.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:59.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:11:59.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:11:59.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:00.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:00.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:00.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:00.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:01.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:01.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:01.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:01.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:02.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:02.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:02.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:02.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:03.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:03.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:03.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:03.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:03.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:03.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:04.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:04.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:05.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:05.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:05.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:05.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:06.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:06.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:06.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:06.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:07.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:07.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:07.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:07.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:08.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:08.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:08.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:08.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:08.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:08.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:09.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:09.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:09.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:09.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:10.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:10.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:10.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:10.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:11.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:11.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:11.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:11.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:12.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:12.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:12.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:12.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:13.374 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:13.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:13.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:13.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:13.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:13.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:14.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:14.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:14.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:14.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:15.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:15.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:15.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:15.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:16.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:16.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:16.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:16.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:17.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:17.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:17.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:17.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:18.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:18.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:18.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:18.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:18.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:18.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:19.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:19.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:19.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:19.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:20.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:20.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:20.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:20.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:21.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:21.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:21.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:21.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:22.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:22.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:22.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:22.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:23.376 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:23.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:23.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:23.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:23.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:23.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:24.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:24.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:24.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:24.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:25.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:25.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:25.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:25.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:26.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:26.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:26.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:26.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:27.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:27.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:27.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:27.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:28.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:28.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:28.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:28.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:28.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:28.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:29.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:29.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:29.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:29.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:30.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:30.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:30.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:30.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:31.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:31.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:31.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:31.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:32.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:32.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:32.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:32.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:33.378 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:33.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:33.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:33.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:33.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:33.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:34.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:34.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:34.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:34.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:35.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:35.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:35.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:35.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:36.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:36.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:36.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:36.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:37.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:37.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:37.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:37.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:38.378 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:38.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:38.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:38.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:38.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:38.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:39.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:39.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:39.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:39.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:40.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:40.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:40.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:40.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:41.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:41.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:41.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:41.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:42.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:42.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:42.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:42.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:43.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:43.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:43.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:43.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:43.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:43.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:44.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:44.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:44.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:44.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:45.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:45.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:45.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:45.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:46.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:46.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:46.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:46.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:47.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:47.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:47.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:47.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:48.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:48.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:48.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:48.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:48.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:48.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:49.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:49.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:49.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:49.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:50.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:50.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:50.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:50.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:51.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:51.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:51.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:51.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:52.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:52.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:52.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:52.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:53.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:53.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:53.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:53.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:53.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:53.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:54.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:54.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:54.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:54.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:55.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:55.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:55.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:55.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:56.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:56.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:56.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:56.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:57.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:57.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:57.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:57.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:58.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:12:58.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:12:58.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:58.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:58.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:58.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:12:59.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:59.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:12:59.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:12:59.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:00.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:00.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:00.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:00.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:01.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:01.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:01.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:01.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:02.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:02.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:02.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:02.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:03.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:03.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:03.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:03.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:03.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:03.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:04.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:04.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:04.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:04.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:05.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:05.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:05.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:05.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:06.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:06.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:06.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:06.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:07.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:07.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:07.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:07.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:08.384 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:08.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:08.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:08.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:08.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:08.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:09.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:09.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:10.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:10.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:10.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:10.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:11.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:11.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:11.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:11.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:12.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:12.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:12.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:12.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:13.384 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:13.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:13.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:13.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:13.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:13.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:14.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:14.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:14.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:14.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:15.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:15.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:15.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:15.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:16.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:16.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:16.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:16.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:17.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:17.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:17.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:17.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:18.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:18.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:18.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:18.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:19.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:19.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:19.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:19.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:20.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:20.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:20.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:20.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:21.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:21.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:21.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:21.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:22.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:22.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:22.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:22.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:23.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:23.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:24.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:23.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:24.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:23.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:25.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:25.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:25.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:25.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:26.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:26.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:26.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:26.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:27.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:27.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:27.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:27.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:28.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:28.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:28.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:28.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:28.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:28.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:29.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:29.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:30.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:30.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:30.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:30.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:31.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:31.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:31.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:31.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:32.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:32.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:32.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:32.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:33.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:33.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:33.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:33.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:33.388 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:33.383+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:34.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:34.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:34.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:34.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:35.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:35.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:35.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:35.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:36.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:36.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:36.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:36.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:37.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:37.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:37.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:37.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:38.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:38.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:38.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:38.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:38.389 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:38.383+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:39.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:39.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:39.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:39.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:40.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:40.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:40.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:40.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:41.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:41.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:41.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:41.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:42.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:42.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:42.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:42.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:43.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:43.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:43.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:43.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:43.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:43.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:44.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:44.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:44.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:44.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:45.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:45.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:45.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:45.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:46.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:46.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:47.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:47.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:47.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:47.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:48.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:48.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:48.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:48.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:48.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:48.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:49.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:49.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:49.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:49.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:50.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:50.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:50.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:50.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:51.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:51.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:51.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:51.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:52.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:52.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:52.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:52.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:53.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:53.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:53.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:53.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:53.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:53.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:54.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:54.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:54.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:54.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:55.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:55.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:55.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:55.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:56.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:56.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:56.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:56.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:57.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:57.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:57.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:57.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:58.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:58.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:58.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:58.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:13:58.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:13:58.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:13:59.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:59.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:13:59.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:13:59.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:00.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:00.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:01.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:01.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:01.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:01.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:02.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:02.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:02.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:02.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:03.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:03.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:03.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:03.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:03.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:03.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:04.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:04.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:04.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:04.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:05.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:05.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:05.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:05.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:06.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:06.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:06.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:06.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:07.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:07.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:07.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:07.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:08.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:08.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:08.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:08.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:08.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:08.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:09.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:09.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:09.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:09.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:10.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:10.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:10.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:11.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:11.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:11.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:11.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:12.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:12.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:12.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:12.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:13.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:13.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:13.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:13.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:13.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:13.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:14.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:14.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:14.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:14.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:15.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:15.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:15.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:15.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:16.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:16.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:16.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:16.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:17.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:17.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:17.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:17.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:18.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:18.385+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:18.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:18.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:18.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:18.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:19.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:19.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:19.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:19.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:20.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:20.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:20.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:20.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:21.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:21.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:21.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:21.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:22.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:22.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:22.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:22.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:23.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:23.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:23.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:23.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:23.397 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:23.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:24.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:24.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:24.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:24.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:25.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:25.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:25.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:25.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:26.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:26.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:27.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:27.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:27.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:27.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:28.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:28.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:28.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:28.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:28.397 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:28.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:29.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:29.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:29.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:29.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:30.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:30.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:30.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:30.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:31.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:31.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:31.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:31.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:32.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:32.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:32.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:32.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:33.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:33.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:33.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:33.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:33.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:33.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:34.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:34.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:34.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:34.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:35.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:35.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:35.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:35.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:36.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:36.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:36.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:36.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:37.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:37.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:37.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:37.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:38.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:38.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:38.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:38.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:38.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:38.393+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:39.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:39.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:39.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:39.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:40.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:40.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:40.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:40.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:41.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:41.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:41.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:41.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:42.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:42.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:42.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:42.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:43.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:43.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:43.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:43.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:43.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:43.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:44.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:44.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:44.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:44.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:45.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:45.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:45.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:45.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:46.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:46.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:46.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:46.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:47.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:47.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:47.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:47.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:48.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:48.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:48.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:48.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:48.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:48.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:49.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:49.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:49.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:49.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:50.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:50.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:50.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:50.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:51.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:51.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:51.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:51.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:52.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:52.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:52.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:52.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:53.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:53.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:53.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:53.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:53.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:53.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:54.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:54.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:54.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:54.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:55.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:55.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:55.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:55.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:56.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:56.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:57.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:57.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:57.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:57.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:58.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:58.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:58.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:58.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:14:58.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:14:58.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:14:59.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:59.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:14:59.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:14:59.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:00.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:15:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:00.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:01.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:01.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:04:26.359831+0000 (oldest deadline 2019-10-17T22:04:46.359831+0000) 2019-10-17T22:15:01.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:01.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:02.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:02.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:03.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:03.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:03.403 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:03.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:04.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:04.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:05.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:05.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:06.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:06.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:07.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:07.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:08.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:08.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:08.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:08.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:09.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:09.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:10.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:10.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:11.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:11.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:12.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:12.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:13.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:13.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:13.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:13.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:14.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:14.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:15.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:15.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:16.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:17.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:17.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:18.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:18.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:18.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:18.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:19.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:19.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:20.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:20.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:21.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:21.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:22.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:22.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:23.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:23.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:23.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:23.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:24.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:24.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:24.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:24.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:25.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:25.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:25.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:25.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:26.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:26.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:26.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:26.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:27.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:27.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:27.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:27.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:28.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:28.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:28.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:28.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:28.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:28.399+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:29.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:29.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:29.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:29.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:30.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:30.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:30.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:30.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:31.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:31.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:31.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:31.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:32.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:32.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:32.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:32.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:33.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:33.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:33.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:33.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:33.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:33.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:34.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:34.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:34.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:34.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:35.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:35.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:35.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:35.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:36.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:36.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:36.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:36.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:37.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:37.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:37.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:37.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:38.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:38.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:38.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:38.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:38.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:38.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:39.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:39.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:39.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:39.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:40.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:40.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:40.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:40.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:41.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:41.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:41.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:41.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:42.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:42.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:42.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:43.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:43.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:43.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:43.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:43.410 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:43.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:44.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:44.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:44.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:44.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:45.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:45.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:46.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:46.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:46.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:46.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:47.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:47.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:47.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:47.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:48.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:48.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:48.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:48.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:48.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:48.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:49.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:49.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:49.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:49.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:50.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:50.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:50.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:50.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:51.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:51.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:51.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:51.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:52.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:52.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:52.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:52.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:53.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:53.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:53.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:53.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:53.412 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:53.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:54.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:54.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:54.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:54.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:55.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:55.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:55.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:55.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:56.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:56.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:56.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:56.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:57.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:57.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:57.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:57.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:58.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:58.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:58.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:58.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:15:58.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:15:58.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:15:59.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:59.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:15:59.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:15:59.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:00.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:00.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:00.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:00.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:01.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:01.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:01.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:01.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:02.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:02.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:02.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:02.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:03.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:03.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:03.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:03.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:03.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:03.406+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:04.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:04.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:04.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:04.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:05.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:05.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:05.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:05.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:06.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:06.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:06.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:06.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:07.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:07.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:07.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:07.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:08.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:08.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:08.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:08.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:08.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:08.406+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:09.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:09.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:09.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:09.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:10.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:10.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:10.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:10.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:11.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:11.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:11.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:11.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:12.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:12.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:12.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:12.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:13.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:13.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:13.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:13.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:13.415 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:13.406+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:14.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:14.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:14.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:14.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:15.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:15.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:15.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:15.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:16.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:16.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:16.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:16.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:17.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:17.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:17.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:17.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:18.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:18.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:18.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:18.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:18.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:18.406+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:19.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:19.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:19.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:19.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:20.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:20.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:20.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:20.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:21.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:21.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:21.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:21.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:22.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:22.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:22.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:22.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:23.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:23.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:23.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:23.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:23.418 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:23.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:24.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:24.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:24.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:24.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:25.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:25.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:25.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:25.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:26.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:26.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:27.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:27.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:27.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:27.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:28.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:28.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:28.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:28.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:28.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:28.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:29.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:29.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:29.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:29.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:30.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:30.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:30.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:30.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:31.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:30.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:31.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:30.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:31.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:31.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:31.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:31.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:32.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:32.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:32.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:32.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:33.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:33.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:33.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:33.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:33.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:33.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:34.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:34.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:35.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:35.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:35.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:35.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:36.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:36.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:36.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:36.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:38.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:37.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:38.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:37.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:38.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:38.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:39.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:39.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:39.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:39.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:40.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:39.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:40.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:39.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:41.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:41.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:41.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:41.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:42.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:42.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:42.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:42.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:43.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:43.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:43.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:43.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:43.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:43.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:44.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:44.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:44.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:44.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:45.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:45.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:45.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:45.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:46.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:46.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:46.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:46.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:47.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:47.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:47.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:47.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:48.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:48.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:48.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:48.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:48.421 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:48.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:49.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:49.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:49.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:49.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:50.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:50.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:50.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:50.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:51.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:51.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:51.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:51.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:52.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:52.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:52.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:52.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:53.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:53.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:53.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:53.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:53.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:53.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:54.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:54.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:54.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:54.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:55.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:55.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:55.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:55.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:56.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:56.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:56.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:56.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:57.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:57.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:57.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:57.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:58.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:58.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:58.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:58.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:16:58.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:16:58.417+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:16:59.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:59.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:16:59.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:16:59.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:00.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:00.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:00.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:00.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:01.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:01.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:01.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:01.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:02.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:02.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:02.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:02.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:03.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:03.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:03.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:03.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:03.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:03.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:04.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:04.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:04.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:04.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:05.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:05.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:05.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:05.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:06.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:06.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:06.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:06.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:07.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:07.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:07.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:07.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:08.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:08.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:08.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:08.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:08.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:08.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:09.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:09.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:09.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:09.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:10.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:10.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:10.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:10.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:11.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:11.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:11.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:11.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:12.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:12.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:12.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:12.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:13.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:13.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:13.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:13.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:13.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:13.420+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:14.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:14.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:14.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:14.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:15.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:15.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:15.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:15.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:16.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:16.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:16.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:16.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:17.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:17.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:17.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:17.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:18.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:18.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:18.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:18.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:18.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:18.420+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:19.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:19.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:19.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:19.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:20.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:20.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:20.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:20.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:21.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:21.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:21.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:21.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:22.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:22.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:22.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:22.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:23.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:23.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:23.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:23.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:23.427 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:23.420+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:24.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:24.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:24.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:24.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:25.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:25.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:26.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:26.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:26.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:26.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:27.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:27.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:27.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:27.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:28.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:28.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:28.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:28.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:28.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:28.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:29.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:29.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:29.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:29.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:30.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:30.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:30.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:30.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:31.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:31.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:31.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:31.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:32.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:32.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:32.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:32.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:33.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:33.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:33.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:33.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:33.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:33.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:34.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:34.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:34.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:34.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:35.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:35.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:35.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:35.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:36.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:36.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:36.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:36.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:37.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:37.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:37.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:37.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:38.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:38.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:38.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:38.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:38.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:38.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:39.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:39.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:39.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:39.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:40.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:40.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:40.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:40.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:41.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:41.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:41.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:41.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:42.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:42.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:42.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:42.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:43.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:43.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:43.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:43.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:43.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:43.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:44.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:44.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:44.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:44.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:45.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:45.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:45.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:45.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:46.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:46.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:46.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:46.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:47.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:47.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:47.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:47.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:48.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:48.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:48.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:48.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:48.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:48.423+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:49.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:49.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:49.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:49.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:50.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:50.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:50.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:50.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:51.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:51.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:51.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:51.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:52.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:52.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:52.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:52.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:53.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:53.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:53.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:53.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:53.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:53.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:54.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:54.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:54.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:54.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:55.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:55.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:55.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:55.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:56.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:56.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:56.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:56.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:57.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:57.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:57.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:57.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:58.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:58.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:58.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:58.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:17:58.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:17:58.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:17:59.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:59.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:17:59.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:17:59.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:00.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:00.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:00.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:00.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:01.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:01.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:01.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:01.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:02.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:02.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:02.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:02.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:03.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:03.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:03.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:03.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:03.434 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:03.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:04.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:04.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:04.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:04.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:05.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:05.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:05.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:05.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:06.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:06.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:06.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:06.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:07.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:07.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:07.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:07.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:08.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:08.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:08.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:08.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:08.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:08.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:09.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:09.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:09.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:09.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:10.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:10.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:10.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:10.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:11.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:11.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:11.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:11.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:12.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:12.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:12.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:12.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:13.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:13.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:13.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:13.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:13.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:13.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:14.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:14.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:14.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:14.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:15.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:15.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:15.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:15.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:16.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:16.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:16.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:16.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:17.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:17.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:17.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:17.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:18.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:18.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:18.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:18.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:18.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:18.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:19.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:19.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:19.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:19.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:20.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:20.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:20.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:20.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:21.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:21.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:21.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:21.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:22.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:22.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:22.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:22.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:23.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:23.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:23.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:23.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:23.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:23.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:24.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:24.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:24.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:24.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:25.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:25.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:25.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:25.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:26.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:26.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:26.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:26.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:27.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:27.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:27.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:27.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:28.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:28.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:28.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:28.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:28.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:28.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:29.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:29.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:29.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:29.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:30.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:30.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:30.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:30.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:31.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:31.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:31.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:31.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:32.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:32.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:32.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:32.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:33.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:33.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:33.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:33.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:33.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:33.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:34.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:34.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:34.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:34.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:35.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:35.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:35.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:35.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:36.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:36.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:36.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:36.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:37.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:37.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:37.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:37.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:38.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:38.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:38.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:38.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:38.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:38.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:39.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:39.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:39.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:39.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:40.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:40.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:40.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:40.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:41.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:41.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:41.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:41.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:42.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:42.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:42.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:42.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:43.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:43.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:43.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:43.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:43.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:43.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:44.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:44.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:44.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:44.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:45.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:45.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:45.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:45.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:46.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:46.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:46.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:46.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:47.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:47.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:47.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:47.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:48.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:48.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:48.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:48.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:48.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:48.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:49.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:49.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:49.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:49.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:50.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:50.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:50.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:50.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:51.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:51.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:51.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:51.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:52.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:52.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:52.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:52.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:53.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:53.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:53.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:53.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:53.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:53.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:54.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:54.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:54.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:54.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:55.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:55.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:55.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:55.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:56.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:56.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:56.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:56.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:57.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:57.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:57.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:57.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:58.444 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:18:58.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:18:58.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:58.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:58.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:58.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:18:59.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:59.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:18:59.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:18:59.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:00.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:00.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:00.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:00.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:01.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:01.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:01.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:01.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:02.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:02.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:02.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:02.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:03.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:03.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:03.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:03.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:04.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:04.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:04.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:04.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:05.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:05.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:05.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:05.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:06.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:06.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:06.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:06.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:07.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:07.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:07.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:07.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:08.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:08.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:08.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:08.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:08.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:08.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:09.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:09.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:09.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:09.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:10.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:10.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:10.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:10.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:11.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:11.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:11.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:11.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:12.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:12.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:12.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:12.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:13.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:13.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:13.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:13.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:13.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:13.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:14.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:14.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:14.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:14.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:15.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:15.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:15.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:15.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:16.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:16.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:16.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:16.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:17.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:17.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:17.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:17.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:18.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:18.441+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:18.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:18.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:18.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:18.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:19.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:19.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:19.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:19.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:20.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:20.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:20.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:20.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:21.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:21.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:21.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:21.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:22.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:22.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:22.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:22.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:23.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:23.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:23.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:23.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:23.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:23.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:24.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:24.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:24.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:24.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:25.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:25.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:25.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:25.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:26.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:26.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:26.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:26.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:27.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:27.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:27.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:27.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:28.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:28.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:28.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:28.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:28.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:28.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:29.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:29.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:29.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:29.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:30.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:30.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:30.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:30.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:31.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:31.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:31.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:31.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:32.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:32.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:32.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:32.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:33.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:33.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:33.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:33.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:33.450 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:33.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:34.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:34.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:34.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:34.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:35.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:35.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:35.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:35.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:36.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:36.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:36.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:36.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:37.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:37.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:37.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:37.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:38.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:38.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:38.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:38.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:38.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:38.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:39.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:39.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:39.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:39.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:40.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:40.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:40.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:40.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:41.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:41.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:41.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:41.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:42.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:42.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:42.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:42.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:43.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:43.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:43.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:43.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:43.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:43.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:44.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:44.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:44.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:44.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:45.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:45.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:45.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:45.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:46.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:46.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:46.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:46.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:47.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:47.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:47.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:47.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:48.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:48.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:48.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:48.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:48.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:48.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:49.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:49.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:49.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:49.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:50.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:50.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:50.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:50.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:51.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:51.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:51.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:51.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:52.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:52.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:52.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:52.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:53.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:53.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:53.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:53.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:53.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:53.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:54.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:54.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:54.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:54.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:55.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:55.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:55.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:55.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:56.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:56.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:56.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:56.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:57.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:57.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:57.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:57.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:58.454 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:19:58.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:19:58.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:58.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:58.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:58.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:19:59.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:59.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:19:59.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:19:59.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:00.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:00.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:00.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:00.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:01.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:01.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:01.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:01.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:02.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:02.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:02.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:02.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:03.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:03.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:03.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:03.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:03.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:03.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:04.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:04.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:04.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:04.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:05.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:05.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:05.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:05.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:06.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:06.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:06.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:06.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:07.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:07.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:07.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:07.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:08.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:08.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:08.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:08.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:08.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:08.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:09.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:09.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:09.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:09.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:10.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:10.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:10.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:10.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:11.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:11.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:11.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:11.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:12.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:12.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:12.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:12.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:13.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:13.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:13.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:13.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:13.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:13.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:14.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:14.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:14.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:14.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:15.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:15.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:15.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:15.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:16.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:16.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:16.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:16.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:17.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:17.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:17.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:17.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:18.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:18.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:18.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:18.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:18.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:18.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:19.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:19.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:19.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:19.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:20.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:20.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:20.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:20.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:21.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:21.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:21.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:21.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:22.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:22.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:22.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:22.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:23.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:23.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:23.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:23.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:23.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:23.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:24.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:24.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:24.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:24.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:25.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:25.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:25.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:25.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:26.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:26.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:26.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:26.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:27.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:27.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:27.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:27.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:28.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:28.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:28.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:28.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:28.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:28.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:29.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:29.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:29.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:29.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:30.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:30.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:30.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:30.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:31.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:31.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:31.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:31.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:32.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:32.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:32.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:32.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:33.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:33.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:33.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:33.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:33.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:33.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:34.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:34.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:34.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:34.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:35.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:35.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:35.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:35.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:36.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:36.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:36.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:36.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:37.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:37.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:37.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:37.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:38.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:38.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:38.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:38.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:38.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:38.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:39.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:39.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:39.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:39.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:40.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:40.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:40.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:40.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:41.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:41.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:41.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:41.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:42.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:42.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:42.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:42.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:43.462 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:43.454+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:43.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:43.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:43.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:43.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:44.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:44.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:44.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:44.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:45.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:45.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:45.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:45.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:46.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:46.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:46.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:46.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:47.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:47.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:47.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:47.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:48.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:48.454+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:48.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:48.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:48.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:48.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:49.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:49.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:49.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:49.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:50.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:50.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:50.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:50.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:51.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:51.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:51.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:51.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:52.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:52.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:52.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:52.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:53.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:53.454+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:53.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:53.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:53.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:53.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:54.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:54.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:54.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:54.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:55.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:55.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:55.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:55.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:56.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:56.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:56.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:56.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:57.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:57.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:57.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:57.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:58.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:20:58.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:20:58.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:58.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:58.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:58.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:20:59.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:59.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:20:59.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:20:59.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:00.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:00.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:00.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:00.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:01.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:01.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:01.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:01.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:02.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:02.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:02.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:02.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:03.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:03.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:03.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:03.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:03.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:03.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:04.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:04.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:04.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:04.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:05.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:05.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:05.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:05.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:06.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:06.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:06.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:06.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:07.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:07.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:07.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:07.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:08.467 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:08.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:08.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:08.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:08.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:08.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:09.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:09.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:09.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:09.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:10.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:10.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:10.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:10.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:11.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:11.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:11.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:11.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:12.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:12.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:12.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:12.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:13.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:13.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:13.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:13.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:13.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:13.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:14.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:14.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:14.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:14.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:15.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:15.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:15.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:15.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:16.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:16.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:16.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:16.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:17.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:17.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:17.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:17.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:18.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:18.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:18.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:18.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:18.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:18.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:19.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:19.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:19.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:19.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:20.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:20.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:21.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:21.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:21.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:21.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:22.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:22.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:22.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:22.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:23.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:23.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:23.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:23.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:23.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:23.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:24.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:24.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:24.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:24.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:25.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:25.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:25.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:25.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:26.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:26.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:26.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:26.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:27.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:27.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:27.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:27.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:28.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:28.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:28.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:28.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:28.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:28.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:29.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:29.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:29.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:29.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:30.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:30.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:30.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:30.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:31.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:31.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:31.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:31.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:32.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:32.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:32.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:32.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:33.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:33.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:33.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:33.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:33.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:33.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:34.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:34.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:34.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:34.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:35.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:35.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:35.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:35.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:36.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:36.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:36.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:36.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:37.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:37.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:37.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:37.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:38.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:38.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:38.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:38.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:38.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:38.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:39.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:39.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:39.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:39.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:40.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:40.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:40.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:40.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:41.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:41.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:41.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:41.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:42.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:42.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:42.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:42.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:43.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:43.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:43.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:43.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:44.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:44.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:45.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:45.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:45.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:45.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:46.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:46.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:46.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:46.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:47.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:47.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:47.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:47.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:48.474 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:48.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:48.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:48.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:48.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:48.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:49.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:49.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:49.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:49.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:50.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:50.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:50.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:50.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:51.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:51.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:51.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:51.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:52.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:52.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:52.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:52.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:53.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:53.468+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:53.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:53.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:53.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:53.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:54.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:54.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:54.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:54.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:55.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:55.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:55.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:55.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:56.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:56.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:56.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:56.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:57.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:57.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:57.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:57.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:58.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:21:58.468+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:21:58.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:58.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:58.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:58.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:21:59.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:59.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:21:59.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:21:59.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:00.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:00.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:00.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:00.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:01.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:01.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:01.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:01.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:02.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:02.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:03.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:03.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:03.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:03.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:03.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:03.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:04.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:04.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:04.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:04.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:05.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:05.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:05.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:05.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:06.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:06.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:06.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:06.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:07.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:07.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:07.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:07.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:08.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:08.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:08.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:08.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:08.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:08.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:09.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:09.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:10.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:10.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:10.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:10.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:11.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:11.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:11.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:11.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:12.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:12.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:12.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:12.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:13.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:13.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:13.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:13.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:13.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:13.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:14.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:14.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:14.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:14.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:15.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:15.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:15.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:15.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:16.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:16.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:16.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:16.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:17.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:17.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:17.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:17.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:18.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:18.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:18.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:18.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:18.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:18.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:19.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:19.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:20.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:20.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:20.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:20.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:21.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:21.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:21.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:21.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:22.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:22.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:22.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:22.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:23.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:23.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:23.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:23.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:23.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:23.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:24.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:24.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:24.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:24.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:25.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:25.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:25.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:25.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:26.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:26.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:26.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:26.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:27.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:27.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:27.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:27.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:28.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:28.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:28.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:28.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:28.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:28.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:29.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:29.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:29.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:29.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:30.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:30.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:30.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:30.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:31.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:31.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:31.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:31.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:32.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:32.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:32.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:32.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:33.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:33.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:33.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:33.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:33.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:33.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:34.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:34.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:34.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:34.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:35.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:35.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:35.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:35.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:36.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:36.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:36.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:36.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:37.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:37.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:37.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:37.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:38.484 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:38.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:38.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:38.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:38.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:38.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:39.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:39.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:39.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:39.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:40.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:40.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:40.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:40.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:41.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:41.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:41.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:41.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:42.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:42.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:42.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:42.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:43.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:43.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:43.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:43.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:43.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:43.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:44.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:44.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:44.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:44.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:45.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:45.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:45.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:45.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:46.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:46.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:46.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:46.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:47.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:47.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:47.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:47.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:48.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:48.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:48.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:48.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:48.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:48.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:49.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:49.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:49.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:49.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:50.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:50.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:50.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:50.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:51.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:51.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:51.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:51.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:52.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:52.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:52.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:52.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:53.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:53.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:53.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:53.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:53.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:53.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:54.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:54.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:54.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:54.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:55.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:55.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:55.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:55.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:56.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:56.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:56.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:56.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:57.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:57.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:57.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:57.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:58.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:22:58.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:22:58.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:58.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:58.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:58.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:22:59.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:59.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:22:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:22:59.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:00.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:00.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:00.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:00.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:01.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:01.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:01.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:01.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:02.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:02.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:02.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:02.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:03.488 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:03.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:03.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:03.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:03.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:03.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:04.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:04.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:04.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:04.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:05.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:05.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:05.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:05.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:06.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:06.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:06.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:06.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:07.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:07.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:07.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:07.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:08.489 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:08.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:08.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:08.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:08.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:08.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:09.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:09.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:09.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:09.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:10.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:10.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:10.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:10.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:11.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:11.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:11.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:11.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:12.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:12.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:12.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:12.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:13.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:13.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:13.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:13.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:13.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:13.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:14.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:14.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:14.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:14.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:15.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:15.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:15.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:15.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:16.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:16.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:16.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:16.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:17.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:17.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:17.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:17.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:18.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:18.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:18.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:18.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:18.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:18.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:19.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:19.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:19.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:19.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:20.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:20.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:20.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:20.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:21.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:21.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:21.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:21.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:22.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:22.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:22.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:22.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:23.491 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:23.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:23.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:23.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:23.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:23.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:24.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:24.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:24.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:24.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:25.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:25.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:25.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:25.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:26.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:26.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:26.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:26.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:27.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:27.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:27.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:27.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:28.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:28.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:28.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:28.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:28.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:28.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:29.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:29.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:29.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:29.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:30.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:30.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:30.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:30.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:31.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:31.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:31.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:31.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:32.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:32.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:32.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:32.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:33.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:33.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:33.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:33.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:33.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:33.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:34.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:34.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:34.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:34.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:35.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:35.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:35.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:35.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:36.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:36.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:36.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:36.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:37.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:37.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:37.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:37.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:38.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:38.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:38.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:38.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:38.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:38.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:39.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:39.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:39.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:39.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:40.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:40.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:40.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:40.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:41.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:41.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:41.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:41.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:42.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:42.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:42.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:42.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:43.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:43.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:43.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:43.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:43.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:43.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:44.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:44.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:44.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:44.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:45.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:45.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:45.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:45.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:46.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:46.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:46.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:46.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:47.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:47.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:47.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:47.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:48.497 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:48.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:48.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:48.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:48.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:48.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:49.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:49.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:49.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:49.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:50.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:50.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:50.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:50.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:51.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:51.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:51.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:51.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:52.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:52.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:52.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:52.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:53.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:53.489+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:53.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:53.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:53.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:53.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:54.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:54.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:54.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:54.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:55.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:55.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:55.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:55.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:56.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:56.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:56.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:56.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:57.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:57.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:57.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:57.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:58.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:58.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:58.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:58.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:23:58.497 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:23:58.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:23:59.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:59.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:23:59.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:23:59.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:00.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:00.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:00.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:00.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:01.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:01.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:01.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:01.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:02.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:02.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:02.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:02.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:03.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:03.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:03.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:03.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:03.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:03.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:04.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:04.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:04.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:04.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:05.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:05.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:06.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:06.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:06.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:06.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:07.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:07.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:07.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:07.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:08.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:08.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:08.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:08.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:08.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:08.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:09.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:09.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:09.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:09.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:10.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:10.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:10.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:10.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:11.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:11.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:11.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:11.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:12.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:12.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:12.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:12.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:13.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:13.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:13.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:13.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:13.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:13.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:14.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:14.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:14.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:14.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:15.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:15.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:15.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:15.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:16.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:16.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:16.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:16.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:17.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:17.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:17.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:17.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:18.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:18.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:18.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:18.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:18.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:18.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:19.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:19.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:19.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:19.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:20.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:20.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:20.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:20.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:21.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:21.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:21.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:21.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:22.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:22.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:22.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:22.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:23.502 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:23.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:23.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:23.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:23.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:23.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:24.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:24.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:24.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:24.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:25.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:25.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:25.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:25.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:26.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:26.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:26.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:26.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:27.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:27.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:27.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:27.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:28.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:28.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:28.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:28.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:28.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:28.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:29.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:29.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:29.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:29.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:30.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:30.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:30.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:30.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:31.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:31.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:31.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:31.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:32.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:32.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:32.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:32.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:33.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:33.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:33.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:33.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:33.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:33.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:34.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:34.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:34.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:34.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:35.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:35.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:35.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:35.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:36.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:36.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:36.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:36.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:37.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:37.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:37.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:37.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:38.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:38.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:38.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:38.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:38.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:38.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:39.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:39.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:39.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:39.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:40.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:40.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:40.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:40.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:41.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:41.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:41.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:41.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:42.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:42.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:42.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:42.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:43.506 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:43.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:43.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:43.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:43.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:43.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:44.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:44.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:44.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:44.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:45.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:45.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:45.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:45.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:46.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:46.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:46.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:46.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:47.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:47.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:47.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:47.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:48.506 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:48.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:48.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:48.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:48.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:48.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:49.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:49.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:49.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:49.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:50.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:50.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:50.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:50.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:51.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:51.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:51.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:51.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:52.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:52.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:52.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:52.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:53.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:53.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:53.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:53.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:53.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:53.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:54.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:54.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:54.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:54.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:55.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:55.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:55.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:55.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:56.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:56.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:56.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:56.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:57.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:57.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:57.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:57.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:58.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:24:58.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:24:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:58.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:58.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:24:59.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:59.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:24:59.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:24:59.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:00.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:00.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:00.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:00.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:01.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:01.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:01.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:01.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:02.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:02.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:02.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:02.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:03.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:03.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:03.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:03.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:03.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:03.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:04.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:04.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:04.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:04.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:05.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:05.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:05.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:05.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:06.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:06.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:06.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:06.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:07.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:07.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:07.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:07.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:08.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:08.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:08.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:08.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:08.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:08.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:09.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:09.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:09.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:09.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:10.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:10.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:10.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:10.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:11.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:11.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:11.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:11.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:12.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:12.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:12.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:12.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:13.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:13.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:13.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:13.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:13.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:13.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:14.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:14.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:14.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:14.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:15.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:15.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:15.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:15.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:16.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:16.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:16.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:16.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:17.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:17.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:17.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:17.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:18.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:18.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:18.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:18.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:18.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:18.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:19.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:19.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:19.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:19.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:20.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:20.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:20.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:20.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:21.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:21.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:21.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:21.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:22.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:22.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:22.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:22.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:23.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:23.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:23.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:23.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:23.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:23.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:24.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:24.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:24.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:24.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:25.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:25.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:25.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:25.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:26.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:26.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:26.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:26.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:27.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:27.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:27.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:27.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:28.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:28.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:28.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:28.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:29.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:29.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:29.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:29.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:30.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:30.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:30.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:30.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:31.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:31.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:31.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:31.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:32.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:32.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:32.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:32.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:33.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:33.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:33.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:33.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:15:03.487289+0000 (oldest deadline 2019-10-17T22:15:23.487289+0000) 2019-10-17T22:25:33.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:33.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:34.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:34.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:35.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:35.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:36.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:36.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:37.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:37.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:38.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:38.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:38.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:38.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:39.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:39.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:40.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:40.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:41.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:41.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:42.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:42.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:43.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:43.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:43.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:43.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:44.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:44.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:45.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:45.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:46.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:46.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:47.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:47.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:48.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:48.510+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:48.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:48.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:49.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:49.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:50.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:50.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:51.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:51.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:52.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:52.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:53.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:53.510+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:53.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:53.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:54.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:54.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:55.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:55.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:56.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:56.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:57.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:57.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:25:57.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:57.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:58.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:25:58.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:25:58.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:58.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:25:58.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:58.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:25:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:59.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:25:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:25:59.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:00.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:00.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:00.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:00.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:01.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:01.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:01.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:01.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:02.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:02.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:02.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:02.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:03.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:03.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:03.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:03.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:03.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:03.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:04.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:04.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:04.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:04.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:05.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:05.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:05.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:05.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:06.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:06.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:06.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:06.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:07.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:07.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:07.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:07.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:08.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:08.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:08.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:08.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:08.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:08.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:09.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:09.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:09.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:09.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:10.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:10.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:10.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:10.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:11.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:11.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:11.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:11.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:12.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:12.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:12.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:12.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:13.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:13.513+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:13.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:13.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:13.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:13.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:14.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:14.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:14.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:14.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:15.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:15.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:15.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:15.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:16.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:16.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:16.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:16.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:17.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:17.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:17.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:17.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:18.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:18.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:18.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:18.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:18.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:18.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:19.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:19.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:19.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:19.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:20.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:20.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:21.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:21.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:21.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:21.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:22.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:22.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:22.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:22.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:23.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:23.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:23.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:23.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:23.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:23.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:24.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:24.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:24.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:24.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:25.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:25.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:25.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:25.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:26.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:26.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:26.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:26.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:27.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:27.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:27.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:27.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:28.524 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:28.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:28.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:28.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:28.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:28.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:29.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:29.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:29.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:29.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:30.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:30.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:30.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:30.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:31.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:31.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:31.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:31.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:32.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:32.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:32.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:32.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:33.524 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:33.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:33.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:33.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:33.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:33.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:34.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:34.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:34.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:34.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:35.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:35.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:35.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:35.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:36.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:36.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:36.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:36.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:37.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:37.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:37.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:37.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:38.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:38.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:38.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:38.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:38.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:38.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:39.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:39.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:39.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:39.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:40.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:40.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:40.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:40.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:41.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:41.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:41.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:41.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:42.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:42.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:42.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:42.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:43.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:43.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:43.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:43.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:43.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:43.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:44.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:44.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:44.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:44.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:45.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:45.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:45.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:45.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:46.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:46.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:46.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:46.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:47.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:47.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:47.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:47.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:48.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:48.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:48.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:48.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:48.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:48.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:49.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:49.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:49.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:49.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:50.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:50.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:50.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:50.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:51.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:51.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:51.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:51.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:52.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:52.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:52.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:52.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:53.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:53.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:53.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:53.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:53.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:53.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:54.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:54.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:54.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:54.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:55.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:55.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:55.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:55.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:56.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:56.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:56.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:56.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:57.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:57.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:57.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:57.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:58.529 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:26:58.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:26:58.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:58.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:58.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:58.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:26:59.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:59.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:26:59.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:26:59.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:00.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:00.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:00.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:00.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:01.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:01.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:01.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:01.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:02.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:02.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:02.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:02.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:03.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:03.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:03.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:03.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:03.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:03.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:04.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:04.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:04.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:04.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:05.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:05.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:05.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:05.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:06.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:06.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:06.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:06.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:07.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:07.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:07.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:07.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:08.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:08.523+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:08.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:08.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:08.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:08.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:09.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:09.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:09.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:09.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:10.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:10.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:10.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:10.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:11.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:11.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:11.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:11.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:12.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:12.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:12.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:12.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:13.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:13.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:13.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:13.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:13.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:13.523+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:14.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:14.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:14.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:14.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:15.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:15.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:15.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:15.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:16.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:16.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:16.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:16.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:17.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:17.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:17.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:17.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:18.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:18.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:18.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:18.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:18.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:18.523+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:19.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:19.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:19.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:19.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:20.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:20.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:20.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:20.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:21.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:21.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:21.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:21.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:22.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:22.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:22.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:22.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:23.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:23.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:23.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:23.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:23.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:23.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:24.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:24.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:24.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:24.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:25.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:25.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:25.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:25.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:26.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:26.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:26.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:26.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:27.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:27.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:27.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:27.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:28.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:28.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:28.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:28.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:28.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:28.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:29.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:29.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:29.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:29.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:30.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:30.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:30.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:30.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:31.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:31.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:31.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:31.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:32.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:32.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:32.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:32.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:33.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:33.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:33.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:33.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:33.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:33.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:34.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:34.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:34.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:34.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:35.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:35.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:35.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:35.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:36.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:36.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:36.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:36.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:37.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:37.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:37.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:37.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:38.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:38.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:38.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:38.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:38.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:38.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:39.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:39.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:39.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:39.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:40.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:40.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:40.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:40.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:41.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:41.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:41.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:41.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:42.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:42.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:42.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:42.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:43.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:43.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:43.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:43.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:43.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:43.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:44.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:44.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:44.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:44.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:45.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:45.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:45.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:45.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:46.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:46.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:46.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:46.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:47.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:47.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:47.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:47.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:48.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:48.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:48.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:48.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:48.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:48.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:49.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:49.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:49.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:49.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:50.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:50.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:50.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:50.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:51.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:51.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:51.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:51.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:52.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:52.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:52.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:52.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:53.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:53.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:53.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:53.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:53.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:53.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:54.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:54.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:54.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:54.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:55.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:55.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:55.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:55.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:56.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:56.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:56.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:56.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:57.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:57.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:57.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:57.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:58.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:58.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:58.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:58.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:27:58.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:27:58.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:27:59.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:59.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:27:59.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:27:59.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:00.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:00.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:00.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:00.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:01.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:01.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:01.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:01.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:02.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:02.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:02.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:02.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:03.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:03.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:03.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:03.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:03.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:03.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:04.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:04.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:04.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:04.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:05.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:05.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:05.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:05.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:06.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:06.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:06.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:06.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:07.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:07.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:07.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:07.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:08.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:08.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:08.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:08.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:08.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:08.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:09.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:09.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:09.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:09.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:10.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:10.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:10.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:10.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:11.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:11.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:11.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:11.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:12.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:12.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:12.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:12.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:13.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:13.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:13.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:13.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:13.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:13.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:14.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:14.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:14.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:14.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:15.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:15.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:15.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:15.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:16.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:16.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:16.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:16.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:17.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:17.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:17.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:17.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:18.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:18.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:18.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:18.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:18.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:18.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:19.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:19.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:19.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:19.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:20.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:20.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:20.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:20.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:21.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:21.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:21.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:21.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:22.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:22.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:22.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:22.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:23.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:23.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:23.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:23.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:23.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:23.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:24.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:24.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:24.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:24.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:25.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:25.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:25.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:25.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:26.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:26.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:26.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:26.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:27.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:27.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:27.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:27.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:28.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:28.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:28.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:28.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:28.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:28.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:29.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:29.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:29.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:29.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:30.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:30.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:30.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:30.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:31.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:31.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:31.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:31.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:32.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:32.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:32.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:32.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:33.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:33.537+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:33.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:33.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:33.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:33.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:34.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:34.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:34.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:34.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:35.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:35.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:35.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:35.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:36.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:36.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:36.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:36.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:37.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:37.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:37.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:37.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:38.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:38.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:38.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:38.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:38.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:38.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:39.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:39.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:39.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:39.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:40.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:40.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:40.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:40.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:41.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:41.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:41.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:41.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:42.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:42.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:42.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:42.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:43.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:43.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:43.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:43.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:43.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:43.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:44.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:44.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:44.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:44.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:45.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:45.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:45.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:45.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:46.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:46.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:46.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:46.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:47.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:47.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:47.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:47.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:48.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:48.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:48.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:48.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:48.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:48.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:49.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:49.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:49.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:49.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:50.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:50.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:50.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:50.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:51.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:51.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:51.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:51.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:52.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:52.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:52.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:52.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:53.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:53.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:53.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:53.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:53.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:53.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:54.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:54.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:54.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:54.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:55.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:55.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:55.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:55.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:56.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:56.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:56.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:56.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:57.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:57.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:57.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:57.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:58.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:28:58.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:28:58.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:58.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:58.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:58.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:28:59.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:59.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:28:59.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:28:59.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:00.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:00.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:00.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:00.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:01.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:01.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:01.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:01.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:02.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:02.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:02.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:02.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:03.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:03.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:03.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:03.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:03.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:03.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:04.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:04.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:04.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:04.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:05.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:05.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:05.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:05.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:06.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:06.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:06.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:06.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:07.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:07.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:07.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:07.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:08.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:08.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:08.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:08.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:08.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:08.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:09.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:09.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:09.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:09.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:10.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:10.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:10.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:10.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:11.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:11.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:11.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:11.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:12.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:12.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:12.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:12.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:13.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:13.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:13.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:13.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:13.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:13.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:14.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:14.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:14.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:14.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:15.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:15.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:15.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:15.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:16.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:16.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:16.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:16.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:17.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:17.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:17.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:17.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:18.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:18.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:18.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:18.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:18.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:18.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:19.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:19.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:19.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:19.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:20.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:20.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:20.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:20.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:21.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:21.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:21.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:21.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:22.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:22.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:22.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:22.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:23.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:23.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:23.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:23.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:23.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:23.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:24.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:24.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:24.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:24.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:25.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:25.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:25.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:25.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:26.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:26.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:26.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:26.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:27.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:27.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:27.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:27.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:28.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:28.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:28.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:28.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:28.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:28.547+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:29.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:29.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:29.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:29.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:30.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:30.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:31.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:31.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:31.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:31.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:32.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:32.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:32.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:32.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:33.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:33.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:33.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:33.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:33.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:33.547+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:34.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:34.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:34.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:34.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:35.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:35.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:35.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:35.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:36.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:36.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:36.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:37.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:37.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:37.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:37.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:38.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:38.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:38.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:38.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:38.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:38.547+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:39.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:39.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:39.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:39.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:40.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:40.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:40.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:40.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:41.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:41.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:41.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:41.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:42.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:42.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:42.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:42.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:43.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:43.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:43.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:43.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:43.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:43.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:44.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:44.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:44.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:44.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:45.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:45.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:45.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:45.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:46.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:46.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:46.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:46.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:47.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:47.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:47.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:47.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:48.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:48.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:48.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:48.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:48.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:48.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:49.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:49.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:49.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:49.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:50.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:50.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:50.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:50.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:51.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:51.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:51.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:51.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:52.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:52.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:52.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:52.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:53.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:53.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:53.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:53.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:53.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:53.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:54.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:54.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:54.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:54.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:55.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:55.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:55.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:55.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:56.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:56.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:56.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:56.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:57.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:57.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:57.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:57.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:58.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:58.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:58.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:58.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:29:58.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:29:58.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:29:59.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:59.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:29:59.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:29:59.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:00.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:00.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:00.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:00.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:01.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:01.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:01.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:01.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:02.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:02.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:02.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:02.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:03.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:03.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:03.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:03.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:03.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:03.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:04.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:04.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:04.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:04.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:05.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:05.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:05.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:05.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:06.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:06.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:06.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:06.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:07.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:07.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:07.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:07.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:08.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:08.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:08.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:08.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:08.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:08.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:09.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:09.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:09.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:09.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:10.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:10.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:10.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:11.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:11.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:11.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:11.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:12.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:12.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:12.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:12.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:13.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:13.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:13.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:13.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:13.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:13.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:14.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:14.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:14.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:14.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:15.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:15.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:15.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:15.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:16.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:16.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:16.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:16.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:17.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:17.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:17.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:18.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:18.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:18.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:18.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:18.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:18.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:19.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:19.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:19.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:19.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:20.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:20.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:20.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:20.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:21.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:21.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:21.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:21.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:22.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:22.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:22.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:22.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:23.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:23.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:23.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:23.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:23.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:23.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:24.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:24.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:24.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:24.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:25.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:25.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:25.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:25.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:26.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:26.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:26.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:26.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:27.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:27.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:27.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:27.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:28.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:28.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:28.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:28.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:28.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:28.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:29.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:29.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:29.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:29.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:30.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:30.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:30.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:30.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:31.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:31.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:31.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:31.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:32.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:32.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:32.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:32.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:33.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:33.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:33.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:33.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:33.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:33.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:34.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:34.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:34.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:34.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:35.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:35.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:35.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:35.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:36.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:36.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:36.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:36.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:37.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:37.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:37.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:37.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:38.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:38.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:38.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:38.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:38.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:38.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:39.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:39.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:39.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:39.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:40.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:40.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:40.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:40.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:41.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:41.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:41.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:41.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:42.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:42.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:42.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:42.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:43.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:43.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:43.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:43.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:43.567 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:43.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:44.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:44.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:44.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:44.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:45.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:45.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:45.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:45.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:46.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:46.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:46.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:46.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:47.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:47.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:47.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:47.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:48.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:48.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:48.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:48.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:48.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:48.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:49.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:49.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:49.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:49.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:50.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:50.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:50.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:50.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:51.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:51.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:51.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:51.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:52.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:52.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:52.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:52.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:53.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:53.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:53.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:53.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:53.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:53.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:54.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:54.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:54.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:54.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:55.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:55.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:55.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:55.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:56.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:56.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:56.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:56.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:57.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:57.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:57.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:57.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:58.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:58.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:58.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:58.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:30:58.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:30:58.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:30:59.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:59.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:30:59.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:30:59.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:00.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:00.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:00.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:00.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:01.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:01.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:01.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:01.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:02.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:02.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:02.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:02.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:03.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:03.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:03.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:03.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:03.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:03.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:04.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:04.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:04.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:04.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:05.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:05.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:05.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:05.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:06.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:06.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:07.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:07.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:07.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:07.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:08.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:08.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:08.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:08.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:08.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:08.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:09.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:09.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:09.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:09.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:10.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:10.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:10.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:10.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:11.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:11.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:11.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:11.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:12.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:12.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:12.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:12.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:13.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:13.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:13.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:13.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:13.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:13.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:14.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:14.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:14.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:14.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:15.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:15.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:15.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:15.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:16.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:16.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:16.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:16.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:17.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:17.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:17.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:17.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:18.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:18.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:18.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:18.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:18.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:18.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:19.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:19.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:19.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:19.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:20.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:20.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:21.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:21.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:21.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:21.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:22.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:22.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:22.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:22.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:23.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:23.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:23.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:23.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:23.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:23.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:24.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:24.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:24.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:24.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:25.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:25.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:25.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:25.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:26.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:26.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:27.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:27.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:27.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:27.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:28.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:28.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:28.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:28.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:28.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:28.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:29.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:29.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:29.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:29.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:30.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:30.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:30.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:30.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:31.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:31.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:31.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:31.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:32.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:32.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:32.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:32.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:33.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:33.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:33.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:33.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:33.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:33.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:34.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:34.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:34.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:34.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:35.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:35.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:35.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:35.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:36.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:36.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:36.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:36.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:37.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:37.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:37.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:37.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:38.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:38.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:38.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:38.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:38.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:38.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:39.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:39.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:39.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:39.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:40.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:40.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:40.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:40.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:41.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:41.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:41.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:41.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:42.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:42.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:42.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:42.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:43.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:43.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:43.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:43.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:43.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:43.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:44.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:44.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:44.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:44.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:45.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:45.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:45.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:45.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:46.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:46.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:46.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:46.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:47.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:47.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:47.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:47.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:48.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:48.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:48.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:48.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:48.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:48.571+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:49.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:49.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:49.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:49.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:50.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:50.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:50.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:50.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:51.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:51.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:51.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:51.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:52.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:52.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:52.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:52.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:53.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:53.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:53.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:53.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:53.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:53.571+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:54.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:54.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:54.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:54.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:55.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:55.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:55.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:55.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:56.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:56.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:56.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:56.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:57.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:57.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:57.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:57.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:58.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:58.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:58.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:58.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:31:58.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:31:58.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:31:59.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:59.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:31:59.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:31:59.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:00.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:00.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:00.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:00.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:01.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:01.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:01.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:01.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:02.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:02.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:02.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:02.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:03.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:03.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:03.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:03.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:03.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:03.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:04.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:04.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:05.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:05.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:05.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:05.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:06.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:06.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:06.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:06.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:07.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:07.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:07.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:07.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:08.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:08.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:08.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:08.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:08.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:08.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:09.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:09.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:09.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:09.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:10.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:10.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:10.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:10.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:11.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:11.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:11.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:11.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:12.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:12.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:12.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:12.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:13.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:13.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:13.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:13.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:13.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:13.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:14.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:14.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:14.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:14.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:15.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:15.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:15.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:15.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:16.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:16.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:16.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:16.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:17.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:17.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:17.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:17.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:18.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:18.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:18.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:18.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:18.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:18.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:19.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:19.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:19.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:19.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:20.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:20.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:20.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:20.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:21.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:21.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:21.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:21.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:22.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:22.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:22.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:22.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:23.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:23.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:23.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:23.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:23.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:23.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:24.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:24.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:24.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:24.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:25.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:25.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:25.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:25.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:26.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:26.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:26.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:26.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:27.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:27.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:27.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:27.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:28.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:28.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:28.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:28.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:28.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:28.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:29.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:29.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:29.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:29.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:30.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:30.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:30.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:30.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:31.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:31.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:31.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:31.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:32.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:32.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:32.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:32.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:33.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:33.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:33.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:33.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:33.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:33.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:34.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:34.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:34.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:34.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:35.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:35.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:35.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:35.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:36.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:36.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:36.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:36.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:37.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:37.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:37.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:37.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:38.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:38.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:38.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:38.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:38.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:38.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:39.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:39.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:39.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:39.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:40.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:40.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:40.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:40.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:41.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:41.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:41.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:41.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:42.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:42.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:42.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:42.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:43.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:43.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:43.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:43.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:43.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:43.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:44.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:44.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:44.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:44.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:45.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:45.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:45.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:45.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:46.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:46.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:46.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:46.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:47.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:47.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:47.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:47.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:48.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:48.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:48.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:48.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:48.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:48.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:49.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:49.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:49.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:49.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:50.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:50.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:50.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:50.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:51.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:51.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:51.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:51.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:52.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:52.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:52.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:52.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:53.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:53.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:53.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:53.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:53.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:53.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:54.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:54.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:54.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:54.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:55.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:55.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:55.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:55.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:56.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:56.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:56.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:56.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:57.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:57.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:57.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:57.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:58.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:58.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:58.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:58.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:32:58.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:32:58.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:32:59.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:59.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:32:59.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:32:59.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:00.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:00.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:00.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:00.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:01.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:01.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:01.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:01.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:02.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:02.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:02.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:02.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:03.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:03.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:03.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:03.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:03.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:03.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:04.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:04.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:04.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:04.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:05.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:05.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:05.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:05.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:06.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:06.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:06.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:06.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:07.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:07.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:07.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:07.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:08.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:08.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:08.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:08.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:08.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:08.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:09.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:09.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:09.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:09.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:10.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:10.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:11.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:11.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:11.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:11.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:12.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:12.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:12.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:12.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:13.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:13.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:13.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:13.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:13.593 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:13.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:14.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:14.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:15.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:15.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:15.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:15.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:16.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:16.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:16.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:16.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:17.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:17.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:17.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:17.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:18.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:18.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:18.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:18.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:18.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:18.589+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:19.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:19.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:19.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:19.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:20.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:20.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:20.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:20.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:21.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:21.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:21.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:21.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:22.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:22.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:22.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:22.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:23.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:23.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:23.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:23.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:23.595 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:23.589+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:24.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:24.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:24.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:24.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:25.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:25.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:25.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:25.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:26.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:26.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:26.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:26.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:27.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:27.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:27.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:27.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:28.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:28.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:28.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:28.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:28.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:28.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:29.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:29.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:29.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:29.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:30.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:30.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:30.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:30.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:31.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:31.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:31.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:31.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:32.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:32.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:32.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:32.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:33.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:33.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:33.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:33.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:33.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:33.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:34.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:34.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:34.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:34.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:35.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:35.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:35.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:35.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:36.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:36.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:36.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:36.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:37.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:37.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:37.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:37.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:38.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:38.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:38.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:38.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:38.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:38.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:39.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:39.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:39.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:39.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:40.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:40.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:40.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:40.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:41.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:41.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:41.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:41.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:42.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:42.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:42.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:42.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:43.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:43.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:43.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:43.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:43.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:43.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:44.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:44.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:44.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:44.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:45.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:45.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:45.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:45.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:46.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:46.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:47.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:47.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:47.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:47.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:48.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:48.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:48.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:48.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:48.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:48.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:49.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:49.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:49.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:49.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:50.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:50.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:50.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:50.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:51.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:51.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:51.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:51.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:52.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:52.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:52.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:52.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:53.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:53.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:53.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:53.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:53.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:53.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:54.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:54.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:54.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:54.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:55.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:55.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:55.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:55.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:56.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:56.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:56.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:56.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:57.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:57.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:57.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:57.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:58.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:58.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:58.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:58.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:33:58.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:33:58.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:33:59.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:59.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:33:59.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:33:59.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:00.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:00.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:00.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:00.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:01.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:01.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:01.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:01.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:02.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:02.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:02.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:02.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:03.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:03.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:03.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:03.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:03.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:03.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:04.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:04.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:04.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:04.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:05.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:05.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:05.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:05.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:06.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:06.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:06.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:06.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:07.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:07.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:07.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:07.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:08.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:08.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:08.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:08.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:08.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:08.595+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:09.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:09.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:09.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:09.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:10.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:10.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:11.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:11.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:11.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:11.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:12.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:12.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:12.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:12.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:13.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:13.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:13.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:13.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:13.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:13.595+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:14.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:14.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:14.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:14.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:15.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:15.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:15.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:15.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:16.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:16.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:16.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:16.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:17.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:17.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:17.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:17.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:18.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:18.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:18.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:18.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:18.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:18.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:19.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:19.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:19.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:19.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:20.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:20.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:20.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:20.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:21.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:21.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:21.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:21.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:22.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:22.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:22.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:22.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:23.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:23.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:23.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:23.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:23.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:23.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:24.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:24.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:24.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:24.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:25.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:25.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:25.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:25.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:26.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:26.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:26.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:26.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:27.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:27.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:27.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:27.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:28.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:28.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:28.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:28.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:28.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:28.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:29.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:29.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:29.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:29.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:30.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:30.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:30.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:30.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:31.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:31.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:31.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:31.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:32.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:32.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:32.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:32.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:33.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:33.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:33.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:33.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:33.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:33.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:34.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:34.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:34.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:34.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:35.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:35.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:35.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:35.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:36.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:36.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:36.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:36.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:37.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:37.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:37.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:37.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:38.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:38.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:38.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:38.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:38.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:38.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:39.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:39.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:39.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:39.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:40.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:40.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:40.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:40.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:41.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:41.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:41.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:41.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:42.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:42.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:42.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:42.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:43.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:43.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:43.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:43.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:43.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:43.602+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:44.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:44.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:44.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:44.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:45.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:45.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:45.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:45.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:46.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:46.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:46.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:46.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:47.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:47.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:47.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:47.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:48.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:48.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:48.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:48.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:48.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:48.602+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:49.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:49.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:49.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:49.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:50.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:50.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:50.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:50.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:51.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:51.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:51.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:51.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:52.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:52.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:52.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:52.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:53.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:53.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:53.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:53.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:53.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:53.602+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:54.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:54.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:55.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:55.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:55.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:55.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:56.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:56.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:56.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:56.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:57.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:57.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:57.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:57.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:58.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:58.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:58.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:58.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:34:58.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:34:58.602+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:34:59.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:59.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:34:59.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:34:59.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:00.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:00.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:00.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:00.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:01.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:01.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:01.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:01.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:02.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:02.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:02.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:02.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:03.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:03.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:03.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:03.602+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:04.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:04.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:04.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:04.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:05.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:05.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:05.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:05.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:06.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:06.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:06.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:06.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:07.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:07.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:07.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:07.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:08.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:08.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:08.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:08.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:09.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:09.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:09.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:09.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:10.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:10.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:10.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:10.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:11.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:11.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:11.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:11.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:12.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:12.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:12.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:12.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:13.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:13.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:13.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:13.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:13.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:13.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:14.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:14.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:14.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:14.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:15.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:15.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:15.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:16.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:16.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:16.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:16.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:17.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:17.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:17.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:17.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:18.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:18.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:18.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:18.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:18.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:18.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:19.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:19.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:20.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:20.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:20.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:20.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:21.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:21.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:21.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:21.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:22.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:22.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:22.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:22.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:23.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:23.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:23.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:23.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:23.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:23.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:24.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:24.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:24.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:24.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:25.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:25.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:25.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:25.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:26.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:26.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:26.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:26.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:27.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:27.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:27.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:27.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:28.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:28.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:28.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:28.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:28.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:28.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:29.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:29.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:29.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:29.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:30.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:30.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:30.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:30.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:31.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:31.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:31.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:31.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:32.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:32.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:32.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:32.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:33.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:33.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:33.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:33.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:33.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:33.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:34.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:34.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:34.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:34.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:35.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:35.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:35.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:35.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:36.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:36.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:36.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:36.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:37.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:37.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:37.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:37.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:38.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:38.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:38.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:38.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:38.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:38.613+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:39.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:39.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:39.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:39.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:40.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:40.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:40.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:40.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:41.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:41.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:41.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:41.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:42.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:42.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:42.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:42.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:43.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:43.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:43.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:43.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:43.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:43.613+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:44.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:44.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:44.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:44.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:45.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:45.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:45.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:45.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:46.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:46.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:46.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:46.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:47.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:47.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:47.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:47.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:48.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:48.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:48.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:48.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:48.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:48.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:49.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:49.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:49.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:49.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:50.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:50.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:50.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:50.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:51.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:51.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:51.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:51.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:52.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:52.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:52.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:52.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:53.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:53.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:53.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:53.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:53.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:53.616+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:54.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:54.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:54.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:54.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:55.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:55.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:55.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:55.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:55.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:55.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:55.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:55.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:56.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:56.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:56.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:56.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:58.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:57.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:58.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:57.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:35:58.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:35:58.616+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:35:59.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:59.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:35:59.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:35:59.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:00.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:00.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:00.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:00.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:01.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:01.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:01.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:01.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:02.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:02.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:02.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:03.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:03.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:03.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:03.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:03.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:03.616+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:04.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:04.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:04.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:04.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:05.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:05.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:05.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:05.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:06.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:06.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:06.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:06.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:07.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:07.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:07.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:07.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:08.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:08.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:08.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:08.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:08.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:08.616+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:09.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:09.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:09.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:09.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:10.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:10.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:10.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:10.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:11.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:11.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:25:37.315378+0000 (oldest deadline 2019-10-17T22:25:57.315378+0000) 2019-10-17T22:36:11.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:11.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:12.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:12.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:13.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:13.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:13.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:13.616+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:14.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:14.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:15.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:15.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:16.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:16.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:17.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:17.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:18.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:18.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:18.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:18.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:19.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:19.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:20.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:20.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:21.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:21.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:22.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:22.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:23.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:23.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:23.626 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:23.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:24.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:24.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:25.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:25.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:26.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:26.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:27.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:27.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:28.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:28.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:28.626 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:28.619+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:29.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:29.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:30.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:30.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:31.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:31.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:32.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:32.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:32.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:32.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:33.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:33.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:33.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:33.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:33.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:33.619+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:34.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:34.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:34.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:34.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:35.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:35.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:35.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:35.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:36.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:36.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:36.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:36.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:37.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:37.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:37.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:37.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:38.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:38.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:38.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:38.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:38.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:38.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:39.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:39.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:39.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:39.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:40.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:40.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:40.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:40.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:41.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:41.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:41.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:41.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:42.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:42.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:42.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:42.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:43.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:43.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:43.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:43.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:43.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:43.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:44.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:44.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:44.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:44.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:45.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:45.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:45.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:45.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:46.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:46.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:46.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:46.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:47.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:47.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:47.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:47.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:48.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:48.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:48.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:48.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:48.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:48.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:49.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:49.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:49.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:49.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:50.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:50.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:50.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:50.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:51.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:51.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:51.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:51.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:52.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:52.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:52.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:52.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:53.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:53.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:53.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:53.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:53.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:53.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:54.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:54.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:54.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:54.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:55.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:55.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:55.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:55.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:56.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:56.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:56.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:56.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:57.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:57.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:57.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:58.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:58.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:58.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:58.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:36:58.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:36:58.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:36:59.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:59.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:36:59.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:36:59.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:00.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:00.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:00.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:00.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:01.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:01.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:01.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:01.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:02.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:02.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:02.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:02.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:03.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:03.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:03.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:03.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:03.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:03.626+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:04.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:04.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:04.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:04.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:05.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:05.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:06.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:06.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:06.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:06.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:07.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:07.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:07.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:07.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:08.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:08.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:08.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:08.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:08.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:08.626+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:09.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:09.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:09.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:09.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:10.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:10.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:10.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:10.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:11.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:11.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:11.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:11.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:12.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:12.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:12.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:12.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:13.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:13.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:13.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:13.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:13.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:13.626+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:14.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:14.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:15.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:15.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:15.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:15.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:16.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:15.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:16.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:15.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:17.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:17.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:17.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:17.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:18.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:18.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:18.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:18.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:18.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:18.626+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:19.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:19.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:19.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:19.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:20.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:20.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:20.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:20.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:20.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:20.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:20.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:20.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:22.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:22.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:22.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:22.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:23.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:22.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:23.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:22.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:23.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:23.626+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:24.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:24.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:24.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:24.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:25.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:25.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:25.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:25.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:26.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:26.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:26.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:26.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:27.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:27.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:27.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:27.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:28.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:28.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:28.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:28.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:28.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:28.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:29.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:29.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:29.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:29.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:30.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:30.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:30.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:30.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:31.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:31.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:31.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:31.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:32.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:32.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:32.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:32.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:33.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:33.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:33.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:33.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:33.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:33.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:34.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:34.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:35.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:35.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:35.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:35.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:36.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:36.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:36.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:36.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:37.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:37.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:37.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:37.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:38.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:38.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:38.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:38.629+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:39.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:39.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:39.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:39.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:40.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:40.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:40.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:40.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:41.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:41.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:41.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:41.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:42.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:42.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:42.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:42.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:43.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:43.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:43.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:43.629+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:44.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:44.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:44.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:44.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:45.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:45.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:45.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:45.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:46.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:46.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:46.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:46.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:47.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:47.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:47.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:47.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:48.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:48.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:48.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:48.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:48.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:48.629+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:49.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:49.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:49.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:49.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:50.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:50.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:50.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:50.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:51.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:51.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:51.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:51.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:52.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:52.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:52.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:52.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:53.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:53.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:53.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:53.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:53.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:53.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:54.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:54.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:54.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:54.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:55.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:55.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:55.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:55.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:56.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:56.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:56.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:56.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:57.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:57.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:57.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:57.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:58.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:58.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:58.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:58.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:37:58.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:37:58.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:37:59.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:59.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:37:59.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:37:59.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:00.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:00.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:00.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:00.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:01.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:01.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:01.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:01.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:02.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:02.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:02.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:02.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:03.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:03.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:03.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:03.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:03.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:03.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:04.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:04.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:04.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:04.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:05.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:05.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:05.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:05.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:06.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:06.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:06.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:06.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:07.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:07.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:07.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:07.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:08.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:08.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:08.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:08.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:08.644 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:08.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:09.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:09.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:09.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:09.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:10.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:10.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:10.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:10.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:11.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:11.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:11.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:11.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:12.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:12.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:12.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:12.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:13.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:13.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:13.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:13.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:13.644 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:13.640+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:14.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:14.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:14.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:14.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:15.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:15.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:15.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:15.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:16.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:16.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:16.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:16.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:17.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:17.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:17.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:17.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:18.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:18.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:18.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:18.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:18.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:18.640+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:19.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:19.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:19.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:19.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:20.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:20.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:20.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:20.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:21.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:21.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:21.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:21.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:22.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:22.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:22.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:22.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:23.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:23.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:23.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:23.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:23.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:23.640+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:24.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:24.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:24.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:24.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:25.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:25.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:25.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:25.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:26.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:26.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:26.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:26.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:27.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:27.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:27.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:27.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:28.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:28.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:28.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:28.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:28.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:28.640+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:29.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:29.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:29.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:29.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:30.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:30.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:30.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:30.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:31.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:31.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:31.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:31.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:32.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:32.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:32.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:32.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:33.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:33.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:33.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:33.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:33.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:33.640+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:34.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:34.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:34.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:34.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:35.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:35.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:35.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:35.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:36.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:36.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:36.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:36.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:37.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:37.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:37.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:37.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:38.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:38.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:38.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:38.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:38.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:38.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:39.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:39.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:39.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:39.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:40.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:40.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:40.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:40.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:41.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:41.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:41.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:41.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:42.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:42.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:42.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:42.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:43.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:43.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:43.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:43.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:43.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:43.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:44.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:44.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:44.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:44.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:45.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:45.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:45.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:45.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:46.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:46.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:46.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:46.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:47.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:47.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:47.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:47.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:48.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:48.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:48.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:48.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:48.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:48.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:49.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:49.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:49.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:49.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:50.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:50.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:50.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:50.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:51.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:51.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:51.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:51.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:52.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:52.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:52.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:52.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:53.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:53.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:53.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:53.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:53.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:53.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:54.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:54.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:54.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:54.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:55.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:55.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:55.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:55.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:56.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:56.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:56.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:56.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:57.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:57.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:57.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:57.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:58.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:58.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:58.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:58.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:38:58.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:38:58.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:38:59.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:59.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:38:59.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:38:59.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:00.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:00.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:00.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:00.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:01.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:01.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:01.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:01.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:02.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:02.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:02.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:02.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:03.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:03.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:03.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:03.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:03.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:03.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:04.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:04.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:04.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:04.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:05.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:05.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:05.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:05.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:06.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:06.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:06.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:06.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:07.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:07.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:07.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:07.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:08.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:08.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:08.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:08.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:08.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:08.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:09.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:09.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:09.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:09.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:10.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:10.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:10.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:10.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:11.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:11.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:11.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:11.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:12.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:12.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:12.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:12.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:13.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:13.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:13.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:13.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:13.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:13.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:14.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:14.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:14.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:14.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:15.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:15.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:15.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:15.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:16.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:16.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:16.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:16.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:17.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:17.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:17.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:17.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:18.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:18.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:18.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:18.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:18.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:18.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:19.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:19.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:19.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:19.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:20.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:20.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:20.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:20.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:21.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:21.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:21.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:21.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:22.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:22.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:22.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:22.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:23.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:23.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:23.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:23.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:23.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:23.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:24.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:24.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:24.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:24.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:25.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:25.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:26.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:26.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:26.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:26.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:27.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:27.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:27.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:27.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:28.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:28.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:28.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:28.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:28.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:28.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:29.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:29.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:29.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:29.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:30.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:30.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:30.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:30.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:31.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:31.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:31.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:31.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:32.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:32.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:32.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:32.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:33.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:33.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:33.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:33.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:33.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:33.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:34.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:34.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:34.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:34.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:35.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:35.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:35.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:35.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:36.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:36.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:36.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:36.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:37.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:37.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:37.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:37.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:38.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:38.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:38.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:38.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:38.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:38.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:39.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:39.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:39.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:39.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:40.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:40.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:40.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:40.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:41.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:41.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:42.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:42.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:42.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:42.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:43.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:43.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:43.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:43.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:43.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:43.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:44.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:44.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:44.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:44.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:45.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:45.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:45.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:45.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:46.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:46.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:46.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:46.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:47.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:47.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:47.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:47.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:48.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:48.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:48.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:48.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:48.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:48.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:49.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:49.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:49.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:49.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:50.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:50.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:50.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:50.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:51.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:51.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:51.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:51.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:52.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:52.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:52.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:52.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:53.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:53.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:53.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:53.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:53.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:53.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:54.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:54.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:54.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:54.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:55.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:55.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:55.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:55.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:56.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:56.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:56.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:56.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:57.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:57.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:57.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:57.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:58.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:58.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:39:58.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:39:58.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:39:59.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:59.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:39:59.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:39:59.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:00.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:00.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:00.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:00.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:01.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:01.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:01.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:01.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:02.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:02.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:02.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:02.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:03.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:03.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:03.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:03.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:03.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:03.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:04.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:04.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:04.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:04.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:05.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:05.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:05.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:05.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:06.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:05.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:06.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:05.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:07.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:07.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:07.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:07.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:08.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:08.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:08.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:08.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:08.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:08.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:09.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:09.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:09.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:09.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:10.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:10.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:10.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:10.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:10.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:10.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:10.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:10.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:11.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:11.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:11.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:11.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:12.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:12.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:13.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:13.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:14.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:14.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:14.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:14.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:15.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:15.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:15.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:15.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:16.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:16.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:16.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:16.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:17.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:17.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:17.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:17.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:18.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:18.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:18.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:18.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:18.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:18.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:19.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:19.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:19.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:19.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:20.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:20.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:20.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:20.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:21.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:21.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:21.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:21.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:22.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:22.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:22.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:22.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:23.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:23.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:23.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:23.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:23.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:23.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:24.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:24.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:24.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:24.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:25.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:25.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:25.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:26.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:26.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:26.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:26.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:27.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:27.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:27.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:27.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:28.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:28.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:28.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:28.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:28.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:28.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:29.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:29.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:29.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:29.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:30.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:30.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:30.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:30.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:31.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:31.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:31.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:31.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:32.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:32.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:32.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:32.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:33.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:33.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:33.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:33.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:33.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:33.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:34.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:34.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:34.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:34.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:35.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:35.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:35.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:35.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:36.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:36.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:36.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:36.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:37.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:37.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:37.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:37.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:38.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:38.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:38.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:38.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:38.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:38.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:39.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:39.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:39.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:39.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:40.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:40.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:40.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:40.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:41.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:41.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:41.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:41.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:42.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:42.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:42.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:42.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:43.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:43.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:43.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:43.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:43.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:43.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:44.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:44.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:44.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:44.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:45.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:45.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:45.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:45.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:46.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:46.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:46.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:46.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:47.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:47.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:47.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:47.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:48.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:48.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:48.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:48.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:48.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:48.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:49.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:49.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:49.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:49.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:50.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:50.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:50.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:50.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:51.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:51.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:51.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:51.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:52.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:52.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:52.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:52.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:53.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:53.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:53.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:53.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:53.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:53.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:54.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:54.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:54.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:54.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:55.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:55.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:55.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:55.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:56.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:56.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:56.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:56.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:57.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:57.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:57.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:58.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:58.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:58.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:58.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:40:58.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:40:58.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:40:59.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:59.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:40:59.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:40:59.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:00.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:00.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:00.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:00.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:01.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:01.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:01.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:01.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:02.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:02.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:02.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:02.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:03.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:03.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:03.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:03.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:03.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:03.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:04.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:04.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:04.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:04.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:05.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:05.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:06.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:06.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:07.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:07.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:07.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:07.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:08.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:08.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:08.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:08.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:08.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:08.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:09.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:09.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:09.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:09.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:10.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:10.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:10.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:10.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:11.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:11.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:11.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:11.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:12.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:12.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:12.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:12.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:13.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:13.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:13.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:13.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:13.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:13.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:14.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:14.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:14.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:14.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:15.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:15.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:15.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:15.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:16.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:16.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:16.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:16.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:17.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:17.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:17.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:17.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:18.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:18.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:18.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:18.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:18.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:18.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:19.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:19.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:19.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:19.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:20.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:20.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:21.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:21.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:21.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:21.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:22.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:22.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:22.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:22.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:23.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:23.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:23.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:23.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:23.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:24.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:24.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:24.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:24.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:25.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:25.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:25.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:25.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:26.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:26.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:26.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:26.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:27.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:27.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:27.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:27.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:28.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:28.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:28.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:28.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:28.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:28.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:29.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:29.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:29.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:29.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:30.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:30.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:30.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:30.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:31.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:31.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:31.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:31.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:32.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:32.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:32.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:32.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:33.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:33.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:33.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:33.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:33.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:33.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:34.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:34.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:34.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:34.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:35.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:35.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:35.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:35.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:36.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:36.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:36.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:36.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:37.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:37.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:37.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:37.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:38.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:38.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:38.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:38.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:38.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:38.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:39.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:39.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:39.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:39.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:40.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:40.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:40.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:40.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:41.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:41.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:41.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:41.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:42.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:42.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:42.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:42.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:43.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:43.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:43.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:43.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:43.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:43.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:44.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:44.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:44.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:44.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:45.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:45.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:45.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:45.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:46.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:46.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:46.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:46.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:47.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:47.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:47.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:47.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:48.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:48.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:48.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:48.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:48.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:48.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:49.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:49.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:49.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:49.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:50.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:50.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:50.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:50.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:51.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:51.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:51.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:51.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:52.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:52.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:52.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:52.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:53.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:53.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:53.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:53.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:53.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:53.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:54.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:54.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:54.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:55.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:55.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:55.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:55.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:56.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:56.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:56.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:56.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:57.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:57.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:57.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:57.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:58.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:58.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:58.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:58.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:41:58.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:41:58.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:41:59.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:59.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:41:59.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:41:59.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:00.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:00.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:00.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:00.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:01.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:01.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:01.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:01.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:02.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:02.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:02.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:02.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:03.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:03.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:03.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:03.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:03.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:03.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:04.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:04.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:04.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:04.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:05.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:05.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:05.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:05.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:06.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:06.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:06.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:06.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:07.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:07.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:07.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:07.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:08.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:08.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:08.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:08.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:08.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:08.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:09.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:09.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:09.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:09.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:10.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:10.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:10.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:10.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:11.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:11.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:11.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:11.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:12.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:12.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:12.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:12.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:13.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:13.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:13.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:13.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:13.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:13.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:14.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:14.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:14.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:14.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:15.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:15.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:15.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:15.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:16.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:16.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:16.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:16.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:17.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:17.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:17.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:17.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:18.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:18.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:18.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:18.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:18.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:18.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:19.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:19.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:19.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:19.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:20.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:20.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:20.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:20.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:21.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:21.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:21.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:21.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:22.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:22.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:22.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:22.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:23.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:23.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:23.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:23.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:23.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:23.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:24.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:24.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:24.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:24.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:25.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:25.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:25.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:25.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:26.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:26.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:26.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:26.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:27.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:27.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:27.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:27.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:28.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:28.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:28.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:28.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:28.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:28.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:29.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:29.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:29.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:29.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:30.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:30.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:30.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:30.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:31.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:31.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:31.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:31.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:32.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:32.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:32.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:32.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:33.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:33.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:33.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:33.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:33.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:33.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:34.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:34.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:34.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:34.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:35.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:35.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:35.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:35.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:36.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:36.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:36.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:36.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:37.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:37.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:37.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:37.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:38.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:38.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:38.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:38.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:38.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:38.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:39.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:39.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:39.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:39.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:40.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:40.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:40.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:40.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:41.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:41.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:41.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:41.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:42.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:42.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:42.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:42.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:43.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:43.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:43.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:43.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:43.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:43.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:44.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:44.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:44.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:44.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:45.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:45.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:45.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:45.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:46.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:46.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:46.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:46.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:47.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:47.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:47.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:47.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:48.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:48.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:48.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:48.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:48.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:48.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:49.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:49.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:49.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:49.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:50.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:50.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:50.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:50.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:51.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:51.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:51.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:51.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:52.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:52.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:52.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:52.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:53.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:53.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:53.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:53.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:53.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:53.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:54.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:54.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:54.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:54.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:55.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:55.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:55.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:55.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:56.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:56.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:56.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:56.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:57.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:57.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:57.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:57.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:58.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:58.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:58.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:58.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:42:58.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:42:58.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:42:59.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:59.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:42:59.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:42:59.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:00.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:00.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:00.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:00.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:01.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:01.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:01.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:01.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:02.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:02.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:02.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:02.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:03.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:03.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:03.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:03.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:03.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:03.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:04.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:04.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:04.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:04.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:05.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:05.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:05.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:05.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:06.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:06.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:06.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:06.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:07.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:07.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:07.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:07.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:08.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:08.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:08.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:08.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:08.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:08.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:09.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:09.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:09.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:09.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:10.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:10.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:10.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:10.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:11.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:11.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:11.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:11.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:12.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:12.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:12.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:12.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:13.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:13.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:13.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:13.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:13.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:13.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:14.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:14.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:14.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:14.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:15.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:15.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:15.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:15.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:16.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:16.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:16.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:16.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:17.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:17.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:17.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:18.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:18.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:18.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:18.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:18.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:18.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:19.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:19.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:19.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:19.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:20.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:20.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:21.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:21.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:21.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:21.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:22.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:22.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:22.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:22.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:23.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:23.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:23.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:23.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:23.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:23.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:24.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:24.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:24.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:24.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:25.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:25.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:25.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:25.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:26.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:26.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:26.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:26.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:27.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:27.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:27.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:27.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:28.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:28.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:28.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:28.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:28.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:28.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:29.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:29.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:29.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:29.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:30.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:30.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:30.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:30.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:31.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:31.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:31.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:31.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:32.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:32.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:32.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:32.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:33.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:33.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:33.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:33.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:33.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:33.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:34.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:34.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:34.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:34.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:35.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:35.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:35.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:35.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:36.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:36.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:36.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:36.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:37.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:37.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:37.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:37.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:38.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:38.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:38.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:38.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:38.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:38.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:39.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:39.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:39.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:39.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:40.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:40.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:40.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:40.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:41.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:41.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:41.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:41.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:42.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:42.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:43.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:43.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:43.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:43.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:43.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:43.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:44.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:44.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:44.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:44.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:45.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:45.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:45.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:45.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:46.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:46.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:46.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:46.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:47.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:47.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:47.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:47.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:48.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:48.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:48.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:48.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:49.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:49.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:49.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:49.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:50.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:50.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:50.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:50.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:51.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:51.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:51.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:51.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:52.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:52.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:52.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:52.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:53.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:53.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:53.703 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:53.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:54.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:54.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:54.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:54.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:55.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:55.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:55.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:55.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:56.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:56.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:56.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:56.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:57.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:57.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:58.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:58.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:58.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:58.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:43:58.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:43:58.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:43:59.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:59.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:43:59.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:43:59.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:00.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:00.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:00.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:00.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:01.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:01.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:01.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:01.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:02.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:02.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:02.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:02.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:03.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:03.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:03.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:03.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:03.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:03.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:04.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:04.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:04.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:04.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:05.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:05.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:05.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:05.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:06.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:06.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:06.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:06.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:07.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:07.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:07.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:07.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:08.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:08.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:08.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:08.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:08.705 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:08.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:09.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:09.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:09.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:09.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:10.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:10.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:10.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:10.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:11.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:11.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:11.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:11.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:12.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:12.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:12.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:12.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:13.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:13.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:13.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:13.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:13.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:13.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:14.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:14.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:14.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:14.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:15.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:15.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:15.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:15.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:16.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:16.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:16.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:16.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:17.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:17.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:17.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:17.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:18.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:18.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:18.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:18.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:18.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:18.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:19.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:19.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:19.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:19.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:20.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:20.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:20.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:20.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:21.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:21.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:21.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:22.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:22.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:22.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:22.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:23.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:23.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:23.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:23.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:23.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:23.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:24.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:24.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:24.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:24.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:25.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:25.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:25.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:25.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:26.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:26.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:26.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:26.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:27.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:27.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:27.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:27.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:28.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:28.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:28.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:28.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:28.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:28.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:29.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:29.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:29.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:29.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:30.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:30.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:30.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:30.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:31.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:31.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:31.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:31.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:32.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:32.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:32.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:32.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:33.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:33.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:33.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:33.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:33.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:33.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:34.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:34.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:34.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:34.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:35.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:35.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:35.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:35.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:36.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:36.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:36.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:36.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:37.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:37.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:37.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:37.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:38.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:38.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:38.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:38.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:38.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:38.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:39.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:39.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:39.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:39.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:40.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:40.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:40.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:40.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:41.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:41.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:41.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:41.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:42.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:42.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:42.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:42.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:43.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:43.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:43.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:43.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:44.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:44.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:44.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:44.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:45.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:45.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:45.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:45.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:46.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:46.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:46.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:46.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:47.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:47.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:47.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:47.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:48.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:48.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:48.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:48.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:48.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:48.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:49.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:49.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:50.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:50.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:50.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:50.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:51.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:51.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:51.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:51.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:52.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:52.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:52.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:52.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:53.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:53.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:53.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:53.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:53.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:53.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:54.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:54.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:54.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:54.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:55.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:55.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:55.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:55.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:56.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:56.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:56.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:56.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:57.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:57.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:57.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:57.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:58.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:58.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:58.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:58.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:44:58.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:44:58.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:44:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:59.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:44:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:44:59.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:00.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:00.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:00.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:00.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:01.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:01.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:01.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:01.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:02.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:02.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:02.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:02.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:03.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:03.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:03.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:03.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:03.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:03.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:04.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:04.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:04.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:04.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:05.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:05.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:05.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:05.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:06.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:06.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:06.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:06.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:07.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:07.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:07.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:07.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:08.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:08.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:08.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:08.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:08.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:08.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:09.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:09.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:09.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:09.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:10.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:10.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:10.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:10.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:11.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:11.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:11.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:11.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:12.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:12.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:12.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:12.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:13.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:13.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:13.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:13.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:13.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:13.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:14.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:14.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:14.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:14.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:15.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:15.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:15.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:15.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:16.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:16.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:16.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:16.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:17.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:17.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:17.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:17.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:18.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:18.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:18.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:18.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:18.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:18.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:19.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:19.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:19.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:19.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:20.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:20.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:20.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:20.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:21.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:21.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:21.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:21.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:22.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:22.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:22.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:22.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:23.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:23.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:23.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:23.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:23.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:23.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:24.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:24.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:24.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:24.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:25.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:25.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:25.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:25.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:26.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:26.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:26.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:27.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:27.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:27.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:27.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:28.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:28.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:28.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:28.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:28.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:28.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:29.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:29.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:29.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:29.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:30.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:30.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:30.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:30.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:31.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:31.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:31.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:31.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:32.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:32.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:32.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:32.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:33.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:33.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:33.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:33.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:33.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:33.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:34.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:34.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:34.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:34.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:35.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:35.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:35.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:35.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:36.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:36.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:36.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:36.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:37.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:37.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:37.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:37.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:38.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:38.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:38.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:38.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:38.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:38.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:39.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:39.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:39.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:39.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:40.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:40.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:40.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:40.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:41.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:41.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:41.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:41.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:42.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:42.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:42.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:42.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:43.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:43.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:43.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:43.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:43.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:43.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:44.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:44.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:44.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:44.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:45.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:45.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:45.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:45.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:46.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:46.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:46.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:46.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:47.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:47.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:47.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:47.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:48.723 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:48.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:48.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:48.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:48.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:48.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:49.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:49.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:49.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:49.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:50.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:50.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:50.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:50.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:51.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:51.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:51.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:51.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:52.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:52.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:52.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:52.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:53.724 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:53.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:53.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:53.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:53.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:53.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:54.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:54.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:54.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:54.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:55.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:55.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:55.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:55.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:56.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:56.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:56.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:56.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:57.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:57.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:57.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:58.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:45:58.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:45:58.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:58.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:58.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:58.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:45:59.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:59.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:45:59.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:45:59.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:00.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:00.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:00.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:00.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:01.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:01.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:01.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:01.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:02.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:02.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:02.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:02.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:03.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:03.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:03.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:03.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:03.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:03.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:04.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:04.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:04.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:04.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:05.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:05.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:05.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:05.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:06.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:06.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:06.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:06.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:07.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:07.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:07.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:07.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:08.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:08.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:08.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:08.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:08.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:08.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:09.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:09.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:09.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:09.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:10.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:10.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:10.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:10.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:11.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:11.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:11.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:11.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:12.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:12.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:12.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:12.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:13.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:13.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:13.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:13.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:13.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:13.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:14.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:14.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:14.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:14.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:15.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:15.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:15.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:15.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:16.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:16.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:17.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:17.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:17.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:17.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:18.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:18.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:18.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:18.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:18.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:18.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:19.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:19.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:19.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:19.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:20.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:20.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:20.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:20.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:21.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:21.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:21.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:21.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:22.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:22.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:22.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:22.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:23.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:23.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:23.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:23.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:23.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:23.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:24.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:24.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:24.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:24.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:25.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:25.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:25.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:25.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:26.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:26.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:26.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:26.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:27.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:27.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:27.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:27.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:28.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:28.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:28.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:28.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:28.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:28.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:29.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:29.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:29.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:29.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:30.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:30.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:30.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:30.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:31.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:31.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:31.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:31.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:32.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:32.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:32.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:32.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:33.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:33.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:33.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:33.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:33.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:33.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:34.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:34.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:34.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:34.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:35.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:35.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:35.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:35.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:36.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:36.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:37.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:37.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:37.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:37.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:38.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:38.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:38.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:38.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:38.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:38.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:39.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:39.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:39.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:39.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:40.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:40.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:40.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:40.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:41.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:41.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:41.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:41.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:42.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:42.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:42.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:42.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:43.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:43.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:43.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:43.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:43.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:43.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:44.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:44.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:44.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:44.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:45.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:45.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:45.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:45.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:46.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:46.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:46.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:46.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:47.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:47.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:47.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:47.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:48.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:48.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:36:11.743919+0000 (oldest deadline 2019-10-17T22:36:31.743919+0000) 2019-10-17T22:46:48.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:48.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:48.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:48.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:49.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:49.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:50.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:50.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:51.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:51.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:52.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:52.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:53.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:53.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:53.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:53.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:54.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:54.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:55.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:55.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:56.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:56.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:57.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:57.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:58.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:58.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:46:58.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:46:58.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:46:59.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:46:59.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:00.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:00.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:01.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:01.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:02.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:02.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:03.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:03.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:03.737 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:03.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:04.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:04.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:05.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:05.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:06.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:06.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:07.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:07.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:08.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:08.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:08.737 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:08.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:09.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:09.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:10.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:10.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:11.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:11.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:12.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:12.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:12.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:12.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:13.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:13.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:13.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:13.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:13.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:13.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:14.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:14.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:14.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:14.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:15.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:15.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:15.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:15.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:16.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:16.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:17.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:17.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:17.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:17.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:18.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:18.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:18.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:18.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:18.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:18.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:19.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:19.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:19.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:19.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:20.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:20.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:20.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:20.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:21.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:21.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:21.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:21.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:22.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:22.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:23.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:23.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:23.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:23.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:23.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:23.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:24.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:24.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:24.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:24.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:25.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:25.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:25.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:25.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:26.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:26.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:26.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:26.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:27.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:27.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:27.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:27.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:28.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:28.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:28.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:28.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:28.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:28.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:29.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:29.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:29.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:29.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:30.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:30.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:30.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:30.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:31.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:31.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:31.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:31.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:32.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:32.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:32.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:32.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:33.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:33.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:33.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:33.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:33.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:33.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:34.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:34.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:34.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:34.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:35.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:35.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:35.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:35.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:36.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:36.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:36.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:36.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:37.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:37.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:37.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:37.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:38.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:38.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:38.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:38.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:38.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:38.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:39.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:39.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:39.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:39.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:40.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:40.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:40.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:40.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:41.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:41.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:41.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:41.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:42.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:42.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:43.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:43.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:43.743 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:43.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:44.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:44.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:44.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:44.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:45.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:45.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:45.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:45.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:46.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:46.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:46.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:46.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:47.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:47.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:47.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:47.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:48.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:48.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:48.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:48.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:48.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:48.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:49.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:49.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:50.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:50.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:50.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:50.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:51.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:51.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:51.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:51.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:52.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:52.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:52.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:52.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:53.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:53.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:53.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:53.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:53.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:53.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:54.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:54.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:54.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:54.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:55.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:55.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:55.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:55.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:56.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:56.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:56.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:56.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:57.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:57.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:57.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:57.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:58.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:58.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:58.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:58.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:47:58.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:47:58.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:47:59.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:59.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:47:59.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:47:59.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:00.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:00.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:01.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:01.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:01.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:01.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:02.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:02.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:02.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:02.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:03.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:03.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:03.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:03.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:04.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:04.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:04.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:04.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:05.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:05.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:05.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:05.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:06.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:06.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:06.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:06.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:07.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:07.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:07.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:07.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:08.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:08.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:08.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:08.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:08.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:08.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:09.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:09.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:09.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:09.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:10.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:10.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:10.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:10.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:11.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:11.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:11.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:11.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:12.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:12.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:12.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:12.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:13.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:13.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:13.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:13.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:13.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:13.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:14.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:14.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:14.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:14.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:15.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:15.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:15.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:15.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:16.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:16.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:16.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:16.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:17.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:17.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:17.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:17.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:18.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:18.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:18.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:18.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:18.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:18.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:19.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:19.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:19.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:19.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:20.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:20.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:21.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:21.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:21.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:21.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:22.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:22.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:22.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:22.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:23.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:23.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:23.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:23.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:23.750 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:23.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:24.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:24.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:24.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:24.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:25.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:25.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:25.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:25.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:26.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:26.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:26.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:26.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:27.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:27.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:27.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:27.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:28.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:28.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:28.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:28.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:28.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:28.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:29.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:29.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:29.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:29.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:30.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:30.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:30.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:30.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:31.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:31.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:31.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:31.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:32.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:32.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:32.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:32.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:33.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:33.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:33.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:33.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:33.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:33.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:34.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:34.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:34.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:34.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:35.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:35.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:35.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:35.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:36.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:36.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:36.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:36.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:37.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:37.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:37.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:37.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:38.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:38.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:38.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:38.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:38.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:38.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:39.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:39.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:39.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:39.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:40.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:40.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:40.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:40.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:41.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:41.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:41.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:41.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:42.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:42.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:42.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:42.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:43.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:43.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:43.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:43.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:43.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:43.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:44.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:44.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:44.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:44.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:45.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:45.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:46.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:46.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:46.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:46.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:47.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:47.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:47.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:47.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:48.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:48.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:48.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:48.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:48.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:48.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:49.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:49.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:49.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:49.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:50.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:50.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:50.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:50.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:51.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:51.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:51.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:51.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:52.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:52.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:52.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:52.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:53.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:53.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:53.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:53.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:53.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:53.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:54.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:54.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:54.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:54.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:55.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:55.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:55.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:55.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:56.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:56.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:56.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:56.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:57.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:57.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:57.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:57.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:58.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:58.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:58.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:58.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:48:58.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:48:58.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:48:59.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:59.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:48:59.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:48:59.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:00.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:00.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:00.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:00.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:01.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:01.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:01.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:02.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:02.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:02.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:02.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:03.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:03.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:03.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:03.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:03.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:03.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:04.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:04.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:04.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:04.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:05.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:05.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:05.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:05.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:06.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:06.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:06.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:06.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:07.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:07.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:07.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:08.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:08.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:08.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:08.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:08.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:08.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:09.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:09.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:09.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:09.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:10.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:10.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:10.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:10.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:11.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:11.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:11.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:11.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:12.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:12.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:12.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:12.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:13.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:13.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:13.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:13.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:13.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:13.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:14.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:14.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:14.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:14.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:14.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:14.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:14.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:14.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:15.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:15.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:15.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:15.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:17.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:17.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:17.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:17.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:17.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:17.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:17.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:17.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:18.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:18.754+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:19.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:18.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:19.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:18.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:20.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:20.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:20.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:20.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:20.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:20.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:20.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:20.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:22.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:22.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:22.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:22.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:23.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:23.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:23.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:23.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:23.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:23.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:24.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:24.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:24.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:24.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:25.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:25.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:25.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:25.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:26.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:26.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:26.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:26.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:27.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:27.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:27.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:27.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:28.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:28.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:28.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:28.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:28.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:28.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:29.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:29.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:29.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:29.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:30.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:30.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:30.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:30.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:31.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:31.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:31.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:31.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:32.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:32.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:32.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:32.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:33.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:33.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:33.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:33.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:33.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:33.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:34.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:34.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:34.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:34.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:35.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:35.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:35.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:35.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:36.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:36.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:36.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:36.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:37.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:37.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:37.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:37.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:38.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:38.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:38.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:38.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:38.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:38.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:39.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:39.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:39.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:39.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:40.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:40.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:40.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:40.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:41.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:41.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:41.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:41.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:42.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:42.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:42.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:42.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:43.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:43.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:43.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:43.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:43.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:43.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:44.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:44.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:44.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:44.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:45.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:45.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:45.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:45.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:46.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:46.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:46.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:46.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:47.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:47.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:47.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:47.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:48.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:48.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:48.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:48.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:48.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:48.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:49.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:49.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:49.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:49.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:50.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:50.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:50.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:50.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:51.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:51.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:51.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:51.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:52.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:52.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:52.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:52.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:53.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:53.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:53.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:53.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:53.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:53.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:54.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:54.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:54.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:54.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:55.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:55.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:55.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:55.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:56.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:56.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:56.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:56.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:57.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:57.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:57.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:57.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:58.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:58.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:58.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:58.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:49:58.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:49:58.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:49:59.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:59.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:49:59.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:49:59.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:00.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:00.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:00.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:00.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:01.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:01.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:01.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:01.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:02.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:02.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:02.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:02.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:03.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:03.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:03.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:03.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:03.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:03.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:04.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:04.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:04.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:04.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:05.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:05.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:05.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:05.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:06.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:06.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:06.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:06.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:07.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:07.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:07.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:07.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:08.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:08.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:08.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:08.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:08.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:08.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:09.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:09.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:09.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:09.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:10.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:10.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:10.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:10.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:11.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:11.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:11.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:11.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:12.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:12.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:12.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:12.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:13.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:13.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:13.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:13.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:13.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:13.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:14.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:14.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:14.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:14.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:15.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:15.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:15.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:15.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:16.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:16.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:16.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:16.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:17.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:17.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:17.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:17.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:18.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:18.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:18.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:18.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:18.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:18.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:19.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:19.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:19.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:19.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:20.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:20.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:20.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:20.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:21.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:21.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:21.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:21.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:22.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:22.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:22.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:22.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:23.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:23.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:23.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:23.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:23.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:23.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:24.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:24.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:24.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:24.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:25.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:25.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:25.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:25.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:26.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:26.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:26.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:26.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:27.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:27.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:27.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:27.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:28.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:28.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:28.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:28.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:28.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:28.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:29.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:29.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:29.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:29.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:30.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:30.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:30.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:30.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:31.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:31.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:31.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:31.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:32.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:32.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:32.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:32.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:33.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:33.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:33.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:33.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:33.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:33.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:34.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:34.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:34.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:34.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:35.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:35.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:35.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:35.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:36.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:36.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:37.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:37.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:37.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:37.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:38.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:38.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:38.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:38.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:38.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:38.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:39.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:39.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:39.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:39.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:40.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:40.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:40.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:40.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:41.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:41.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:41.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:41.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:42.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:42.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:42.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:42.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:43.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:43.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:43.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:43.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:43.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:43.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:44.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:44.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:44.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:44.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:45.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:45.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:45.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:45.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:46.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:46.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:46.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:46.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:47.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:47.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:47.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:47.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:48.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:48.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:48.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:48.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:48.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:48.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:49.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:49.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:49.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:49.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:50.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:50.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:50.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:50.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:51.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:51.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:51.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:51.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:52.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:52.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:52.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:52.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:53.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:53.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:53.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:53.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:53.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:53.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:54.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:54.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:54.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:54.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:55.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:55.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:55.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:55.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:56.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:56.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:56.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:56.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:57.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:57.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:57.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:57.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:58.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:58.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:58.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:58.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:50:58.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:50:58.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:50:59.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:59.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:50:59.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:50:59.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:00.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:00.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:00.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:00.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:01.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:01.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:01.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:01.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:02.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:02.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:02.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:02.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:03.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:03.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:03.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:03.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:03.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:03.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:04.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:04.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:04.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:04.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:05.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:05.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:05.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:05.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:06.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:06.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:06.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:06.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:07.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:07.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:07.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:07.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:08.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:08.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:08.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:08.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:08.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:08.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:09.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:09.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:09.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:09.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:10.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:10.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:10.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:10.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:11.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:11.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:11.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:11.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:12.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:12.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:12.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:12.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:13.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:13.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:13.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:13.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:13.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:13.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:14.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:14.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:14.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:14.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:15.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:15.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:15.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:15.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:16.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:16.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:16.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:16.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:17.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:17.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:17.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:17.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:18.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:18.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:18.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:18.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:18.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:18.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:19.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:19.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:19.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:19.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:20.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:20.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:20.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:20.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:21.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:21.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:21.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:21.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:22.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:22.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:22.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:22.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:23.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:23.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:23.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:23.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:23.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:23.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:24.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:24.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:24.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:24.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:25.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:25.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:25.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:25.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:26.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:26.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:26.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:26.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:27.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:27.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:27.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:27.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:28.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:28.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:28.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:28.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:28.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:28.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:29.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:29.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:29.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:29.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:30.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:30.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:30.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:30.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:31.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:31.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:31.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:31.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:32.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:32.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:32.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:32.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:33.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:33.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:33.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:33.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:33.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:33.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:34.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:34.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:34.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:34.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:35.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:35.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:35.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:35.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:36.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:36.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:36.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:36.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:37.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:37.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:37.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:37.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:38.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:38.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:38.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:38.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:38.790 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:38.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:39.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:39.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:39.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:39.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:40.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:40.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:40.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:40.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:41.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:41.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:41.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:41.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:42.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:42.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:42.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:42.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:43.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:43.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:43.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:43.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:43.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:43.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:44.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:44.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:44.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:44.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:45.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:45.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:45.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:45.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:46.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:46.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:46.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:46.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:47.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:47.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:47.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:47.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:48.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:48.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:48.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:48.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:48.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:48.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:49.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:49.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:49.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:49.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:50.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:50.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:50.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:50.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:51.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:51.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:51.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:51.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:52.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:52.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:52.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:52.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:53.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:53.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:53.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:53.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:53.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:53.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:54.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:54.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:54.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:54.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:55.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:55.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:55.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:55.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:56.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:56.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:56.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:56.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:57.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:57.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:57.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:57.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:58.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:58.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:58.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:58.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:51:58.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:51:58.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:51:59.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:59.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:51:59.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:51:59.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:00.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:00.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:00.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:00.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:01.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:01.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:01.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:01.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:02.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:02.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:02.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:02.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:03.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:03.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:03.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:03.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:04.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:04.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:04.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:04.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:05.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:05.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:05.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:05.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:06.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:06.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:06.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:06.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:07.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:07.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:07.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:07.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:08.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:08.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:08.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:08.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:08.790 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:08.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:09.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:09.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:09.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:09.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:10.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:10.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:10.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:10.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:11.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:11.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:11.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:11.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:12.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:12.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:12.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:12.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:13.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:13.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:13.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:13.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:13.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:13.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:14.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:14.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:14.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:14.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:15.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:15.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:15.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:15.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:16.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:16.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:16.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:16.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:17.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:17.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:17.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:17.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:18.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:18.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:18.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:18.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:18.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:18.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:19.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:19.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:19.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:19.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:20.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:20.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:20.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:20.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:21.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:21.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:21.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:21.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:22.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:22.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:22.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:22.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:23.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:23.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:23.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:23.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:23.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:23.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:24.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:24.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:24.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:24.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:25.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:25.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:25.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:25.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:26.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:26.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:26.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:26.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:27.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:27.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:27.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:27.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:28.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:28.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:28.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:28.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:28.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:28.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:29.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:29.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:29.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:29.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:30.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:30.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:30.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:30.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:31.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:31.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:31.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:31.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:32.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:32.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:32.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:32.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:33.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:33.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:33.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:33.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:33.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:33.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:34.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:34.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:34.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:34.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:35.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:35.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:35.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:35.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:36.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:36.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:36.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:36.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:37.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:37.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:37.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:37.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:38.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:38.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:38.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:38.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:38.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:38.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:39.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:39.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:39.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:39.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:40.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:40.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:40.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:40.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:41.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:41.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:42.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:42.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:42.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:42.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:43.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:43.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:43.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:43.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:43.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:43.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:44.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:44.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:44.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:44.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:45.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:45.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:45.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:45.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:46.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:46.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:46.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:46.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:47.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:47.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:47.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:47.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:48.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:48.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:48.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:48.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:48.797 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:48.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:49.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:49.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:49.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:49.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:50.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:50.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:50.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:50.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:51.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:51.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:51.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:51.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:52.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:52.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:52.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:52.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:53.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:53.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:53.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:53.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:53.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:53.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:54.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:54.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:54.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:54.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:55.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:55.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:55.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:55.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:56.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:56.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:56.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:56.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:57.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:57.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:57.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:57.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:58.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:58.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:58.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:58.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:52:58.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:52:58.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:52:59.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:59.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:52:59.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:52:59.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:00.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:00.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:00.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:00.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:01.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:01.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:01.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:01.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:02.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:02.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:02.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:02.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:03.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:03.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:03.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:03.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:03.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:03.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:04.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:04.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:04.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:04.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:05.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:05.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:05.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:05.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:06.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:06.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:06.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:06.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:07.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:07.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:07.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:07.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:08.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:08.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:08.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:08.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:08.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:08.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:09.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:09.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:09.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:09.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:10.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:10.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:10.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:10.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:11.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:11.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:11.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:11.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:12.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:12.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:12.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:12.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:13.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:13.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:13.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:13.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:13.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:13.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:14.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:14.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:14.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:14.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:15.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:15.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:15.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:15.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:16.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:16.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:16.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:16.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:17.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:17.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:17.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:17.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:18.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:18.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:18.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:18.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:19.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:19.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:19.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:19.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:20.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:20.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:21.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:21.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:21.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:21.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:22.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:22.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:22.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:22.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:23.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:23.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:23.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:23.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:23.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:23.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:24.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:24.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:24.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:24.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:25.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:25.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:25.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:25.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:26.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:26.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:26.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:26.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:27.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:27.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:27.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:27.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:28.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:28.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:28.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:28.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:28.803 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:28.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:29.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:29.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:29.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:29.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:30.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:30.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:30.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:30.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:31.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:31.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:31.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:31.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:32.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:32.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:32.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:32.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:33.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:33.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:33.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:33.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:33.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:33.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:34.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:34.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:34.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:34.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:35.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:35.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:35.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:35.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:36.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:36.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:36.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:36.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:37.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:37.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:37.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:37.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:38.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:38.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:38.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:38.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:38.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:38.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:39.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:39.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:39.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:39.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:40.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:40.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:40.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:40.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:41.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:41.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:41.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:41.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:42.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:42.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:42.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:42.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:43.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:43.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:43.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:43.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:43.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:43.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:44.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:44.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:44.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:44.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:45.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:45.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:45.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:45.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:46.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:46.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:46.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:46.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:47.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:47.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:47.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:47.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:48.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:48.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:48.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:48.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:48.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:48.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:49.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:49.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:49.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:49.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:50.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:50.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:50.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:50.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:51.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:51.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:51.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:51.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:52.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:52.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:52.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:52.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:53.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:53.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:53.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:53.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:54.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:54.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:54.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:54.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:55.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:55.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:55.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:55.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:56.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:56.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:56.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:56.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:57.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:57.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:57.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:57.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:58.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:58.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:58.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:58.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:53:58.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:53:58.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:53:59.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:59.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:53:59.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:53:59.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:00.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:00.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:00.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:00.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:01.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:01.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:01.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:01.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:02.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:02.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:02.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:02.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:03.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:03.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:03.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:03.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:03.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:03.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:04.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:04.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:04.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:04.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:05.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:05.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:05.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:05.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:06.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:06.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:06.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:06.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:07.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:07.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:07.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:07.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:08.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:08.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:08.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:08.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:08.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:08.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:09.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:09.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:09.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:09.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:10.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:10.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:10.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:10.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:11.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:11.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:11.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:11.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:12.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:12.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:12.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:12.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:13.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:13.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:13.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:13.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:13.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:13.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:14.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:14.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:14.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:14.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:15.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:15.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:15.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:15.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:16.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:16.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:16.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:16.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:17.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:17.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:17.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:17.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:18.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:18.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:18.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:18.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:18.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:18.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:19.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:19.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:19.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:19.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:20.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:20.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:20.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:20.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:21.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:21.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:21.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:21.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:22.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:22.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:22.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:22.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:23.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:23.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:23.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:23.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:23.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:23.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:24.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:24.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:24.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:25.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:25.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:25.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:25.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:26.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:26.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:26.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:26.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:27.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:27.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:27.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:27.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:28.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:28.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:28.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:28.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:28.814 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:28.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:29.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:29.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:29.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:29.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:30.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:30.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:30.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:30.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:31.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:31.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:31.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:31.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:32.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:32.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:32.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:32.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:33.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:33.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:33.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:33.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:33.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:33.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:34.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:34.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:34.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:34.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:35.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:35.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:35.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:35.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:36.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:36.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:36.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:36.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:37.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:37.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:37.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:37.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:38.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:38.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:38.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:38.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:38.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:38.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:39.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:39.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:39.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:39.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:40.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:40.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:40.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:40.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:41.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:41.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:42.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:42.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:42.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:42.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:43.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:43.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:43.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:43.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:43.816 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:43.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:44.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:44.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:44.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:44.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:45.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:45.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:45.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:46.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:46.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:46.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:46.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:47.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:47.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:47.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:47.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:48.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:48.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:48.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:48.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:48.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:48.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:49.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:49.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:49.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:49.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:50.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:50.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:50.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:50.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:51.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:51.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:51.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:51.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:52.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:52.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:52.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:52.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:53.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:53.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:53.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:53.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:53.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:53.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:54.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:54.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:54.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:54.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:55.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:55.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:55.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:55.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:56.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:56.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:57.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:57.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:57.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:57.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:58.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:58.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:58.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:58.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:54:58.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:54:58.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:54:59.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:59.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:54:59.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:54:59.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:00.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:00.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:00.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:00.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:01.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:01.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:01.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:01.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:02.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:02.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:02.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:02.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:03.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:03.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:03.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:03.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:03.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:03.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:04.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:04.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:04.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:04.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:05.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:05.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:05.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:05.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:06.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:06.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:06.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:06.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:07.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:07.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:07.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:07.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:08.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:08.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:08.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:08.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:08.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:08.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:09.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:09.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:09.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:09.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:10.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:10.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:10.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:10.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:11.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:11.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:11.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:11.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:12.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:12.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:12.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:12.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:13.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:12.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:13.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:12.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:13.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:13.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:14.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:14.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:14.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:14.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:15.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:14.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:15.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:14.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:16.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:16.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:16.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:16.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:17.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:17.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:17.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:17.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:18.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:18.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:18.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:18.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:18.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:18.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:19.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:18.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:19.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:18.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:20.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:20.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:20.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:20.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:21.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:20.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:21.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:20.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:22.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:21.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:22.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:21.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:23.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:23.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:23.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:23.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:23.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:23.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:23.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:23.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:23.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:23.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:25.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:25.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:25.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:25.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:26.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:25.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:26.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:25.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:27.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:27.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:27.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:27.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:28.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:28.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:28.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:28.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:28.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:28.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:29.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:29.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:29.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:29.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:30.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:30.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:30.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:30.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:31.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:31.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:31.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:31.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:31.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:31.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:31.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:31.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:32.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:32.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:32.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:32.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:33.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:33.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:33.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:33.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:33.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:33.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:35.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:35.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:35.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:35.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:36.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:36.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:36.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:36.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:37.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:37.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:37.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:37.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:38.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:38.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:38.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:38.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:38.826 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:38.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:39.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:39.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:39.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:39.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:40.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:40.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:40.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:40.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:41.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:41.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:41.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:41.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:42.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:42.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:42.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:42.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:43.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:43.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:43.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:43.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:43.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:43.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:44.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:44.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:44.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:44.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:45.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:45.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:45.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:45.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:46.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:46.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:46.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:46.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:47.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:47.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:47.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:47.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:48.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:48.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:48.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:48.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:48.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:48.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:49.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:49.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:50.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:50.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:50.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:50.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:51.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:51.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:51.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:51.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:52.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:52.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:52.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:52.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:53.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:53.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:53.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:53.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:53.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:53.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:54.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:54.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:54.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:54.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:55.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:55.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:55.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:55.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:56.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:56.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:56.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:56.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:57.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:57.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:57.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:57.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:58.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:58.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:58.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:58.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:55:58.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:55:58.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:55:59.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:59.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:55:59.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:55:59.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:00.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:00.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:00.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:00.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:01.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:01.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:01.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:01.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:02.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:02.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:02.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:02.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:03.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:03.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:03.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:03.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:03.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:03.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:04.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:04.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:04.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:04.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:05.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:05.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:06.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:06.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:06.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:06.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:07.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:07.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:07.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:07.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:08.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:08.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:08.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:08.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:08.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:08.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:09.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:09.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:10.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:10.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:10.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:10.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:11.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:11.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:11.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:11.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:12.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:12.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:12.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:12.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:13.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:13.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:13.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:13.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:13.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:13.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:14.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:14.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:15.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:15.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:15.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:15.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:16.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:16.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:16.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:16.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:17.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:17.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:17.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:17.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:18.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:18.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:18.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:18.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:18.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:18.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:19.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:19.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:19.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:19.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:20.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:20.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:20.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:20.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:21.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:21.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:21.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:21.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:22.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:22.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:22.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:22.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:23.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:23.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:23.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:23.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:23.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:23.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:24.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:24.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:24.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:24.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:25.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:25.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:25.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:25.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:26.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:26.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:26.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:26.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:27.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:27.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:27.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:27.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:28.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:28.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:28.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:28.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:28.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:28.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:29.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:29.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:29.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:29.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:30.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:30.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:30.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:30.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:31.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:31.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:31.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:31.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:32.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:32.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:32.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:32.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:33.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:33.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:33.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:33.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:33.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:33.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:34.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:34.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:34.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:34.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:35.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:35.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:35.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:35.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:36.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:36.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:36.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:36.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:37.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:37.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:37.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:37.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:38.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:38.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:38.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:38.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:38.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:38.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:39.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:39.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:39.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:39.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:40.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:40.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:40.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:40.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:41.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:41.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:42.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:42.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:42.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:42.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:43.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:43.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:43.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:43.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:43.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:43.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:44.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:44.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:44.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:44.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:45.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:45.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:45.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:45.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:46.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:46.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:46.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:46.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:47.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:47.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:47.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:47.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:48.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:48.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:48.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:48.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:48.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:48.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:49.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:49.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:49.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:49.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:50.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:50.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:50.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:50.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:51.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:51.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:51.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:51.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:52.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:52.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:52.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:52.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:53.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:53.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:53.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:53.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:53.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:53.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:54.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:54.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:54.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:54.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:55.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:55.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:55.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:55.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:56.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:56.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:56.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:56.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:57.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:57.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:57.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:57.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:57.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:57.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:58.839 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:56:58.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:56:58.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:58.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:58.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:58.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:56:59.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:59.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:56:59.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:56:59.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:00.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:00.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:00.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:00.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:01.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:01.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:01.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:01.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:02.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:02.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:02.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:02.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:03.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:03.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:03.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:03.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:03.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:03.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:04.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:04.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:04.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:04.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:05.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:05.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:05.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:05.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:06.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:06.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:06.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:06.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:07.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:07.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:07.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:07.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:08.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:08.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:08.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:08.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:08.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:08.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:09.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:09.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:09.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:09.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:10.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:10.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:10.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:10.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:11.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:11.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:11.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:11.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:12.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:12.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:12.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:12.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:13.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:13.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:13.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:13.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:14.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:14.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:14.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:14.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:15.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:15.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:15.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:15.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:16.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:16.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:16.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:16.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:17.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:17.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:17.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:17.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:18.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:18.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:18.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:18.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:18.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:18.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:19.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:19.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:46:52.269920+0000 (oldest deadline 2019-10-17T22:47:12.269920+0000) 2019-10-17T22:57:19.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:19.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:20.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:20.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:21.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:21.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:22.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:22.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:23.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:23.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:23.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:23.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:24.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:24.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:25.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:25.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:26.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:26.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:27.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:27.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:28.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:28.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:28.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:28.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:29.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:29.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:30.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:30.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:31.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:31.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:32.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:32.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:33.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:33.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:33.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:33.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:34.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:34.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:35.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:35.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:36.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:36.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:37.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:37.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:38.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:38.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:38.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:38.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:39.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:39.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:40.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:40.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:40.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:40.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:41.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:41.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:41.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:41.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:42.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:42.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:42.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:42.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:43.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:43.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:43.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:43.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:43.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:43.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:44.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:44.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:44.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:44.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:45.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:45.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:45.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:45.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:46.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:46.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:46.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:46.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:47.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:47.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:47.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:47.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:48.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:48.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:48.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:48.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:48.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:48.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:49.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:49.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:49.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:49.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:50.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:50.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:50.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:50.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:51.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:51.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:51.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:51.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:52.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:52.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:52.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:52.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:53.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:53.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:53.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:53.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:53.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:53.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:54.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:54.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:54.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:54.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:55.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:55.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:55.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:55.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:56.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:56.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:56.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:56.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:57.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:57.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:57.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:57.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:58.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:58.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:58.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:58.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:57:58.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:57:58.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:57:59.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:59.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:57:59.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:57:59.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:00.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:00.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:00.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:00.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:01.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:01.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:01.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:01.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:02.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:02.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:02.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:02.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:03.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:03.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:03.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:03.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:03.850 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:03.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:04.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:04.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:04.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:04.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:05.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:05.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:05.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:05.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:06.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:06.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:06.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:06.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:07.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:07.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:07.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:07.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:08.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:08.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:08.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:08.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:08.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:08.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:09.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:09.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:09.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:09.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:10.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:10.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:10.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:10.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:11.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:11.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:11.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:11.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:12.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:12.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:12.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:12.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:13.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:13.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:13.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:13.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:13.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:13.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:14.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:14.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:14.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:14.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:15.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:15.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:15.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:15.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:16.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:16.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:16.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:16.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:17.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:17.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:17.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:17.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:18.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:18.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:18.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:18.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:18.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:18.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:19.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:19.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:19.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:20.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:20.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:20.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:20.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:21.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:21.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:21.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:21.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:22.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:22.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:22.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:22.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:23.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:23.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:23.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:23.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:23.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:23.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:24.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:24.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:24.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:24.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:25.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:25.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:25.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:25.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:26.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:26.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:26.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:26.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:27.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:27.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:27.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:27.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:28.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:28.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:28.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:28.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:28.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:28.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:29.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:29.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:29.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:29.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:30.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:30.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:30.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:30.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:31.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:31.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:31.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:31.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:32.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:32.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:32.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:32.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:33.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:33.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:33.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:33.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:33.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:33.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:34.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:34.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:34.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:34.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:35.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:35.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:35.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:35.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:36.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:36.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:36.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:36.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:37.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:37.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:37.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:37.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:38.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:38.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:38.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:38.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:38.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:38.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:39.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:39.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:39.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:39.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:40.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:40.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:40.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:40.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:41.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:41.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:41.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:41.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:42.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:42.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:42.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:42.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:43.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:43.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:43.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:43.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:43.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:43.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:44.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:44.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:44.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:44.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:45.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:45.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:45.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:45.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:46.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:46.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:46.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:46.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:47.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:47.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:47.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:47.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:48.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:48.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:48.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:48.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:48.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:48.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:49.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:49.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:49.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:49.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:50.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:50.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:50.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:50.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:51.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:51.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:51.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:51.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:52.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:52.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:52.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:52.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:53.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:53.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:53.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:53.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:53.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:53.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:54.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:54.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:54.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:54.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:55.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:55.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:55.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:55.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:56.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:56.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:56.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:56.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:57.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:57.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:57.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:57.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:58.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:58:58.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:58:58.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:58.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:58.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:58.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:58:59.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:59.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:58:59.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:58:59.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:00.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:00.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:00.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:00.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:01.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:01.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:01.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:01.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:02.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:02.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:02.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:02.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:03.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:03.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:03.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:03.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:03.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:03.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:04.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:04.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:04.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:04.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:05.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:05.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:05.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:05.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:06.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:06.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:06.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:06.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:07.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:07.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:07.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:07.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:08.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:08.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:08.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:08.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:08.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:08.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:09.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:09.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:09.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:09.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:10.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:10.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:10.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:10.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:11.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:11.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:11.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:11.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:12.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:12.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:12.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:12.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:13.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:13.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:13.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:13.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:13.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:13.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:14.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:14.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:14.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:14.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:15.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:15.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:15.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:15.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:16.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:16.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:16.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:16.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:17.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:17.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:17.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:17.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:18.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:18.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:18.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:18.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:18.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:18.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:19.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:19.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:19.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:19.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:20.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:20.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:20.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:20.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:21.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:21.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:21.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:21.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:22.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:22.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:22.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:22.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:23.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:23.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:23.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:23.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:23.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:23.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:24.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:24.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:24.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:24.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:25.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:25.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:25.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:25.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:26.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:26.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:26.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:26.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:27.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:27.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:27.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:27.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:28.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:28.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:28.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:28.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:28.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:28.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:29.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:29.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:29.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:29.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:30.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:30.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:30.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:30.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:31.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:31.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:31.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:31.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:32.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:32.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:32.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:32.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:33.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:33.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:33.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:33.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:33.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:33.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:34.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:34.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:34.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:34.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:35.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:35.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:35.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:35.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:36.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:36.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:36.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:36.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:37.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:37.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:37.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:37.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:38.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:38.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:38.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:38.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:38.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:38.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:39.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:39.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:39.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:39.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:40.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:40.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:40.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:40.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:41.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:41.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:41.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:41.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:42.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:42.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:43.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:43.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:43.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:43.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:43.868 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:43.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:44.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:44.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:44.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:44.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:45.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:45.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:45.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:45.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:46.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:46.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:46.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:46.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:47.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:47.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:47.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:47.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:48.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:48.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:48.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:48.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:48.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:48.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:49.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:49.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:49.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:49.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:50.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:50.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:50.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:50.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:51.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:51.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:51.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:51.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:52.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:52.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:52.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:52.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:53.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:53.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:53.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:53.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:53.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:53.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:54.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:54.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:54.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:54.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:55.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:55.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:55.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:55.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:56.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:56.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:56.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:56.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:57.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:57.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:57.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:57.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:58.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:58.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:58.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:58.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T22:59:58.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T22:59:58.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T22:59:59.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:59.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T22:59:59.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T22:59:59.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:00.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:00.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:00.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:00.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:01.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:01.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:01.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:01.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:02.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:02.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:02.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:02.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:03.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:03.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:03.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:03.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:03.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:03.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:04.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:04.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:04.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:04.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:05.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:05.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:06.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:06.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:06.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:06.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:07.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:07.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:07.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:07.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:08.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:08.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:08.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:08.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:08.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:08.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:09.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:09.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:09.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:09.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:10.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:10.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:10.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:10.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:11.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:11.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:11.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:11.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:12.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:12.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:12.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:12.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:13.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:13.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:13.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:13.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:13.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:13.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:14.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:14.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:14.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:14.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:15.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:15.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:15.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:15.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:16.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:16.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:16.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:16.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:17.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:17.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:17.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:17.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:18.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:18.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:18.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:18.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:18.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:18.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:19.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:19.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:19.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:19.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:20.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:20.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:20.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:20.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:21.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:21.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:21.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:21.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:22.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:22.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:22.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:22.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:23.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:23.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:23.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:23.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:23.875 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:23.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:24.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:24.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:24.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:24.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:25.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:25.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:25.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:25.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:26.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:26.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:26.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:26.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:27.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:27.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:27.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:27.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:28.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:28.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:28.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:28.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:28.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:28.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:29.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:29.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:29.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:29.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:30.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:30.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:30.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:30.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:31.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:31.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:31.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:31.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:32.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:32.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:32.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:32.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:33.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:33.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:33.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:33.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:33.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:33.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:34.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:34.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:34.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:34.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:35.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:35.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:35.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:35.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:36.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:36.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:36.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:36.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:37.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:37.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:37.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:37.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:38.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:38.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:38.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:38.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:38.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:38.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:39.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:39.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:39.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:39.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:40.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:40.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:40.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:40.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:41.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:41.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:41.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:41.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:42.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:42.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:43.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:43.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:43.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:43.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:44.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:44.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:44.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:44.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:45.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:45.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:45.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:45.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:46.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:46.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:46.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:46.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:47.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:47.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:47.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:47.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:48.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:48.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:48.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:48.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:48.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:48.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:49.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:49.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:49.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:49.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:50.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:50.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:50.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:50.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:51.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:51.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:51.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:51.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:52.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:52.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:52.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:52.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:53.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:53.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:53.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:53.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:53.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:53.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:54.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:54.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:54.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:54.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:55.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:55.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:55.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:55.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:56.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:56.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:56.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:56.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:57.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:57.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:57.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:57.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:58.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:58.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:58.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:58.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:00:58.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:00:58.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:00:59.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:59.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:00:59.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:00:59.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:00.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:00.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:00.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:00.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:01.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:01.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:01.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:01.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:02.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:02.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:02.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:02.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:03.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:03.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:03.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:03.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:03.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:03.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:04.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:04.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:04.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:04.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:05.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:05.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:05.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:05.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:06.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:06.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:06.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:06.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:07.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:07.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:07.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:07.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:08.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:08.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:08.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:08.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:08.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:08.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:09.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:09.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:09.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:09.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:10.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:10.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:10.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:10.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:11.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:11.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:11.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:11.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:12.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:12.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:12.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:12.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:13.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:13.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:13.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:13.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:13.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:13.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:14.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:14.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:14.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:14.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:15.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:15.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:15.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:15.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:16.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:16.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:16.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:16.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:17.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:17.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:17.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:17.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:18.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:18.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:18.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:18.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:18.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:18.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:19.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:19.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:19.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:19.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:20.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:20.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:20.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:20.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:21.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:21.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:21.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:21.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:22.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:22.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:23.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:23.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:23.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:23.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:23.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:23.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:24.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:24.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:24.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:24.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:25.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:25.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:25.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:25.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:26.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:26.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:27.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:27.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:27.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:27.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:28.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:28.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:28.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:28.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:28.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:28.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:29.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:29.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:29.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:29.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:30.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:30.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:30.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:30.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:31.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:31.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:31.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:31.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:32.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:32.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:32.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:32.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:33.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:33.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:33.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:33.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:33.887 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:33.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:34.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:34.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:34.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:34.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:35.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:35.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:35.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:35.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:36.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:36.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:36.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:36.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:37.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:37.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:37.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:37.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:38.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:38.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:38.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:38.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:38.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:38.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:39.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:39.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:39.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:39.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:40.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:40.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:40.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:40.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:41.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:41.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:41.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:41.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:42.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:42.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:42.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:42.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:43.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:43.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:43.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:43.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:43.889 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:43.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:44.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:44.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:44.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:44.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:45.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:45.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:45.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:45.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:46.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:46.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:46.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:46.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:47.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:47.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:47.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:47.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:48.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:48.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:48.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:48.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:48.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:48.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:49.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:49.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:49.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:49.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:50.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:50.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:50.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:50.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:51.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:51.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:51.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:51.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:52.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:52.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:52.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:52.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:53.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:53.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:54.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:53.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:54.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:53.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:54.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:54.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:54.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:54.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:55.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:55.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:55.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:55.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:56.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:56.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:56.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:56.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:57.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:57.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:57.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:57.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:58.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:58.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:58.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:58.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:01:58.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:01:58.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:01:59.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:59.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:01:59.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:01:59.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:00.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:00.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:00.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:00.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:01.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:01.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:01.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:01.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:02.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:02.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:02.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:02.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:03.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:03.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:03.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:03.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:03.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:03.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:04.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:04.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:04.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:04.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:05.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:05.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:06.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:06.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:06.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:06.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:07.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:07.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:07.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:07.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:08.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:08.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:08.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:08.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:08.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:08.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:09.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:09.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:09.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:09.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:10.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:10.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:10.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:10.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:11.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:11.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:11.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:11.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:12.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:12.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:12.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:12.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:13.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:13.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:13.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:13.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:13.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:13.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:14.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:14.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:14.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:14.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:15.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:15.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:15.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:15.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:16.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:16.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:16.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:16.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:17.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:17.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:17.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:17.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:18.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:18.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:18.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:18.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:18.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:18.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:19.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:19.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:19.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:19.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:20.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:20.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:20.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:20.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:21.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:21.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:21.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:21.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:22.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:22.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:22.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:22.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:23.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:23.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:23.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:23.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:23.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:23.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:24.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:24.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:24.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:24.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:25.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:25.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:25.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:25.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:26.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:26.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:26.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:26.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:27.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:27.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:27.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:27.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:28.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:28.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:28.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:28.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:28.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:28.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:29.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:29.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:29.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:29.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:30.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:30.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:30.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:30.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:31.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:31.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:31.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:31.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:32.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:32.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:32.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:32.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:33.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:33.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:33.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:33.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:33.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:33.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:34.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:34.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:34.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:34.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:35.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:35.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:35.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:35.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:36.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:36.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:36.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:36.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:37.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:37.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:37.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:37.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:38.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:38.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:38.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:38.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:38.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:38.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:39.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:39.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:39.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:39.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:40.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:40.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:40.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:40.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:41.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:41.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:42.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:42.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:42.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:42.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:43.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:43.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:43.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:43.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:43.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:43.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:44.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:44.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:44.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:44.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:45.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:45.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:45.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:45.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:46.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:46.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:46.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:46.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:47.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:47.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:47.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:47.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:48.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:48.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:48.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:48.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:48.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:48.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:49.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:49.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:49.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:49.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:50.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:50.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:50.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:50.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:51.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:51.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:51.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:51.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:52.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:52.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:52.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:52.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:53.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:53.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:53.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:53.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:53.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:53.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:54.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:54.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:54.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:54.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:55.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:55.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:55.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:55.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:56.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:56.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:56.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:56.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:57.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:57.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:57.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:57.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:58.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:58.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:02:58.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:02:58.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:02:59.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:59.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:02:59.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:02:59.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:00.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:00.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:00.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:00.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:01.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:01.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:01.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:01.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:02.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:02.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:02.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:02.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:03.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:03.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:03.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:03.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:03.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:03.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:04.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:04.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:04.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:04.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:05.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:05.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:05.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:05.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:06.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:06.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:06.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:06.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:07.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:07.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:07.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:07.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:08.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:08.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:08.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:08.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:08.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:08.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:09.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:09.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:09.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:09.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:10.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:10.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:10.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:10.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:11.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:11.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:11.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:11.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:12.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:12.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:12.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:12.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:13.904 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:13.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:13.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:13.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:13.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:13.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:14.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:14.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:14.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:14.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:15.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:15.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:15.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:15.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:17.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:17.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:17.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:17.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:17.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:17.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:17.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:17.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:18.904 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:18.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:18.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:18.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:18.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:18.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:19.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:19.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:19.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:19.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:20.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:20.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:20.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:20.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:21.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:21.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:21.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:21.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:22.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:22.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:22.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:22.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:23.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:23.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:23.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:23.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:23.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:23.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:24.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:24.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:24.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:24.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:25.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:25.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:25.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:25.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:26.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:26.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:27.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:26.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:28.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:28.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:28.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:28.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:28.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:28.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:28.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:28.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:28.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:28.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:30.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:30.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:30.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:30.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:31.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:31.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:31.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:31.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:32.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:32.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:32.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:32.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:33.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:33.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:33.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:33.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:33.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:33.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:34.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:34.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:34.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:34.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:35.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:35.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:35.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:35.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:36.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:36.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:36.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:36.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:37.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:37.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:37.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:37.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:38.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:38.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:38.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:38.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:38.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:38.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:39.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:39.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:39.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:39.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:40.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:40.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:40.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:40.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:41.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:41.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:41.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:41.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:42.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:42.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:42.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:42.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:43.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:43.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:43.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:43.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:43.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:43.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:44.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:44.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:44.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:44.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:45.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:45.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:45.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:45.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:46.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:46.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:46.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:46.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:47.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:47.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:47.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:47.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:48.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:48.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:48.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:48.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:48.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:48.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:49.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:49.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:49.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:49.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:50.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:50.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:50.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:50.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:51.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:51.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:51.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:51.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:52.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:52.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:52.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:52.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:53.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:53.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:53.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:53.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:53.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:53.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:54.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:54.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:54.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:54.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:55.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:55.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:55.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:55.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:56.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:56.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:56.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:56.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:57.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:57.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:57.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:57.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:58.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:58.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:58.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:58.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:03:58.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:03:58.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:03:59.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:59.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:03:59.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:03:59.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:00.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:00.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:00.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:00.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:01.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:01.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:01.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:01.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:02.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:02.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:02.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:02.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:03.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:03.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:03.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:03.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:03.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:03.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:04.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:04.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:04.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:04.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:05.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:05.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:05.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:05.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:06.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:06.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:06.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:06.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:07.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:07.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:07.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:07.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:08.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:08.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:08.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:08.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:08.913 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:08.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:09.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:09.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:09.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:09.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:10.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:10.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:10.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:10.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:11.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:11.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:11.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:11.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:12.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:12.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:12.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:12.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:13.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:13.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:13.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:13.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:13.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:13.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:14.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:14.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:14.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:14.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:15.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:15.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:15.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:15.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:16.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:16.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:16.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:16.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:17.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:17.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:17.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:18.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:18.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:18.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:18.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:18.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:18.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:19.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:19.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:20.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:20.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:20.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:20.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:21.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:21.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:21.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:21.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:22.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:22.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:22.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:22.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:23.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:23.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:23.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:23.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:23.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:23.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:24.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:24.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:24.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:24.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:25.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:25.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:25.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:25.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:26.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:26.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:26.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:26.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:27.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:27.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:27.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:27.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:28.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:28.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:28.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:28.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:28.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:28.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:29.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:29.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:29.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:29.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:30.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:30.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:30.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:30.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:31.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:31.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:31.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:31.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:32.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:32.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:32.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:32.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:33.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:33.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:33.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:33.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:33.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:33.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:34.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:34.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:34.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:34.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:35.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:35.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:35.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:35.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:36.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:36.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:36.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:36.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:37.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:37.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:37.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:37.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:38.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:38.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:38.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:38.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:38.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:38.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:39.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:39.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:39.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:39.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:40.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:40.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:40.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:40.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:41.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:41.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:41.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:41.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:42.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:42.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:42.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:42.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:43.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:43.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:43.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:43.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:43.919 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:43.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:44.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:44.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:44.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:44.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:45.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:45.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:45.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:45.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:46.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:46.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:46.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:46.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:47.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:47.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:47.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:47.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:48.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:48.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:48.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:48.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:48.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:48.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:49.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:49.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:49.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:49.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:50.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:50.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:50.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:50.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:51.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:51.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:51.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:51.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:52.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:52.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:52.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:52.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:53.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:53.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:53.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:53.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:53.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:53.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:54.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:54.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:54.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:54.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:55.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:55.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:55.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:55.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:56.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:56.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:56.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:56.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:57.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:57.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:57.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:57.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:58.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:58.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:58.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:58.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:04:58.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:04:58.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:04:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:59.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:04:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:04:59.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:00.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:00.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:00.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:00.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:01.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:01.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:01.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:01.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:02.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:02.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:02.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:02.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:03.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:03.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:03.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:03.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:03.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:03.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:04.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:04.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:04.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:04.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:05.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:05.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:05.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:05.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:06.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:06.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:06.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:06.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:07.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:07.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:07.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:07.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:08.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:08.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:08.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:08.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:08.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:08.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:09.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:09.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:09.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:09.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:10.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:10.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:10.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:10.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:11.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:11.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:12.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:12.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:12.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:12.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:13.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:13.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:13.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:13.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:13.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:13.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:14.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:14.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:14.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:14.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:15.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:15.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:15.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:15.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:16.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:16.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:16.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:16.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:17.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:17.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:18.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:18.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:18.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:18.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:18.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:18.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:19.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:19.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:19.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:19.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:20.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:20.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:20.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:20.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:21.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:21.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:21.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:21.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:22.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:22.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:22.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:22.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:23.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:23.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:23.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:23.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:23.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:23.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:24.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:24.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:25.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:25.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:25.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:25.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:26.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:26.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:26.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:26.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:27.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:27.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:27.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:27.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:28.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:28.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:28.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:28.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:28.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:28.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:29.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:29.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:29.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:29.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:30.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:30.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:30.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:30.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:31.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:31.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:31.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:31.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:32.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:32.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:32.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:32.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:33.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:33.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:33.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:33.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:33.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:33.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:34.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:34.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:34.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:34.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:35.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:35.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:35.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:35.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:36.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:36.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:36.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:36.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:37.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:37.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:37.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:37.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:38.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:38.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:38.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:38.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:38.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:38.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:39.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:39.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:39.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:39.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:40.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:40.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:40.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:40.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:41.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:41.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:41.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:41.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:42.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:42.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:42.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:42.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:43.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:43.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:43.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:43.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:43.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:43.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:44.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:44.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:44.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:44.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:45.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:45.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:45.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:45.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:46.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:46.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:46.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:46.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:47.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:47.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:47.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:47.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:48.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:48.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:48.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:48.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:48.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:48.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:49.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:49.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:49.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:49.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:50.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:50.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:50.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:50.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:51.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:51.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:51.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:51.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:52.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:52.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:52.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:52.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:53.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:53.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:53.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:53.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:53.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:53.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:54.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:54.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:54.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:54.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:55.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:55.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:55.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:55.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:56.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:56.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:56.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:56.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:57.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:57.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:57.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:57.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:58.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:58.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:58.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:58.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:05:58.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:05:58.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:05:59.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:59.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:05:59.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:05:59.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:00.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:00.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:00.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:00.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:01.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:01.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:01.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:01.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:02.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:02.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:02.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:02.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:03.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:03.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:03.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:03.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:03.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:03.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:04.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:04.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:04.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:04.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:05.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:05.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:05.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:05.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:06.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:06.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:06.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:06.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:07.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:07.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:07.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:07.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:08.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:08.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:08.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:08.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:08.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:08.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:09.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:09.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:09.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:09.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:10.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:10.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:10.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:10.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:11.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:11.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:11.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:11.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:12.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:12.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:12.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:12.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:13.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:13.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:13.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:13.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:13.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:13.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:14.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:14.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:14.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:14.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:15.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:15.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:15.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:15.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:16.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:16.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:16.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:16.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:17.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:17.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:17.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:17.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:18.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:18.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:18.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:18.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:18.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:18.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:19.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:19.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:19.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:19.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:20.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:20.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:20.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:20.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:21.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:21.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:22.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:22.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:22.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:22.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:23.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:23.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:23.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:23.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:23.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:23.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:25.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:25.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:25.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:25.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:25.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:25.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:25.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:25.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:27.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:27.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:27.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:27.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:27.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:27.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:27.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:27.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:28.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:28.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:28.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:28.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:28.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:28.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:29.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:29.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:29.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:29.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:30.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:30.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:31.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:31.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:31.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:31.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:33.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:33.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:33.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:33.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:33.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:33.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:33.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:33.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:33.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:33.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:34.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:34.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:34.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:34.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:35.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:35.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:35.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:35.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:36.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:36.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:36.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:36.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:37.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:37.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:37.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:37.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:38.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:38.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:38.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:38.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:38.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:38.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:39.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:39.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:39.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:39.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:40.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:40.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:40.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:40.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:41.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:41.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:41.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:41.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:42.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:42.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:42.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:42.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:43.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:43.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:43.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:43.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:43.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:43.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:44.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:44.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:44.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:44.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:45.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:45.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:45.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:45.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:46.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:46.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:46.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:46.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:47.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:47.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:47.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:47.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:48.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:48.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:48.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:48.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:48.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:48.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:49.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:49.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:49.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:49.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:50.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:50.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:50.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:50.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:51.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:51.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:51.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:51.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:52.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:52.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:52.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:52.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:53.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:53.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:53.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:53.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:53.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:53.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:54.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:54.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:54.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:54.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:55.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:55.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:55.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:55.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:56.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:56.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:56.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:56.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:57.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:57.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:57.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:57.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:58.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:58.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:58.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:58.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:06:58.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:06:58.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:06:59.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:59.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:06:59.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:06:59.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:00.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:00.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:00.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:00.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:01.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:01.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:01.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:01.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:02.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:02.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:02.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:02.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:03.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:03.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:03.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:03.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:03.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:03.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:04.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:04.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:04.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:04.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:05.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:05.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:05.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:05.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:06.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:06.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:06.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:06.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:07.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:07.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:07.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:07.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:08.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:08.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:08.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:08.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:08.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:08.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:09.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:09.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:09.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:09.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:10.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:10.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:10.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:10.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:11.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:11.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:11.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:11.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:12.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:12.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:12.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:12.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:13.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:13.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:13.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:13.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:13.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:13.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:14.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:14.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:14.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:14.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:15.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:15.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:15.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:15.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:16.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:16.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:16.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:16.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:17.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:17.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:17.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:17.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:18.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:18.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:18.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:18.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:18.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:18.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:19.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:19.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:19.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:19.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:20.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:20.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:20.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:20.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:22.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:21.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:22.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:21.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:23.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:22.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:23.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:22.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:23.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:23.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:23.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:23.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:23.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:23.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:25.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:25.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:25.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:25.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:26.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:26.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:26.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:26.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:27.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:27.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:27.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:27.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:28.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:28.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:28.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:28.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:28.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:28.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:29.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:29.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:29.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:29.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:30.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:30.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:30.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:30.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:31.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:31.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:31.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:31.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:32.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:32.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:33.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:33.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:33.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:33.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:33.947 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:33.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:34.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:34.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:34.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:34.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:35.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:35.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:35.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:35.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:36.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:36.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:36.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:36.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:37.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:37.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:37.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:37.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:37.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:37.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:37.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:37.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:38.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:38.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:38.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:38.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:38.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:38.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:40.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:40.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:40.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:40.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:40.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:40.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:40.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:40.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:42.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:42.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:42.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:42.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:43.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:43.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:43.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:43.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:43.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:43.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:44.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:44.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:44.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:44.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:45.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:45.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:45.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:45.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:46.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:46.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:46.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:46.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:47.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:47.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:47.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:47.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:48.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:48.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:48.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:48.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:48.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:48.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:49.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:49.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:49.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:49.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:50.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:50.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:50.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:50.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:51.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:51.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:51.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:51.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:52.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:52.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:52.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:52.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:53.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:53.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T22:57:20.397632+0000 (oldest deadline 2019-10-17T22:57:40.397632+0000) 2019-10-17T23:07:53.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:53.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:53.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:53.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:54.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:54.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:55.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:55.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:56.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:56.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:57.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:57.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:58.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:58.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:07:58.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:07:58.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:07:59.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:07:59.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:00.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:00.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:01.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:01.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:02.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:02.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:03.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:03.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:03.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:03.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:04.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:04.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:05.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:05.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:06.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:06.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:07.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:07.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:08.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:08.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:08.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:08.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:09.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:09.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:10.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:10.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:11.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:11.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:12.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:12.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:13.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:13.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:13.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:13.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:14.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:14.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:15.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:15.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:16.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:16.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:17.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:17.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:18.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:18.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:18.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:18.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:19.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:19.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:19.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:19.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:20.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:20.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:20.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:20.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:21.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:21.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:21.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:21.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:22.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:22.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:22.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:22.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:23.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:23.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:23.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:23.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:23.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:23.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:24.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:24.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:24.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:24.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:25.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:25.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:26.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:26.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:26.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:26.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:27.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:27.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:27.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:27.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:28.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:28.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:28.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:28.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:28.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:28.953+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:29.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:29.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:29.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:29.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:30.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:30.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:30.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:30.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:31.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:31.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:31.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:31.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:32.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:32.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:33.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:33.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:33.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:33.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:33.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:33.953+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:34.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:34.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:34.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:35.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:35.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:35.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:35.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:36.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:36.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:36.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:36.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:37.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:37.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:37.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:37.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:38.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:38.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:38.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:38.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:38.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:38.953+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:39.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:39.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:39.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:39.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:40.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:40.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:40.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:40.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:41.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:41.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:41.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:42.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:42.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:42.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:42.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:43.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:43.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:43.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:43.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:43.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:43.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:44.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:44.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:44.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:44.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:45.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:45.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:45.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:45.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:46.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:46.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:46.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:46.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:47.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:47.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:47.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:47.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:48.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:48.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:48.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:48.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:48.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:48.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:49.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:49.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:49.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:49.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:50.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:50.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:50.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:50.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:51.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:51.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:51.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:51.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:52.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:52.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:52.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:52.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:53.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:53.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:53.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:53.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:53.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:53.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:54.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:54.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:54.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:54.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:55.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:55.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:55.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:55.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:56.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:56.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:56.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:56.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:57.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:57.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:57.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:57.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:58.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:58.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:58.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:58.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:08:58.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:08:58.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:08:59.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:59.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:08:59.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:08:59.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:00.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:00.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:00.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:00.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:01.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:01.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:01.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:01.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:02.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:02.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:02.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:02.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:02.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:02.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:02.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:02.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:03.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:03.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:03.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:03.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:03.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:03.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:04.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:04.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:04.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:04.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:05.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:05.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:05.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:05.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:06.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:06.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:06.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:06.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:07.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:07.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:07.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:07.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:08.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:08.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:08.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:08.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:08.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:08.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:09.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:09.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:09.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:09.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:10.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:10.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:10.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:10.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:11.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:11.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:11.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:11.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:13.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:13.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:13.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:13.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:13.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:13.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:13.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:13.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:13.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:13.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:15.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:15.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:15.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:15.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:15.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:15.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:15.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:15.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:16.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:16.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:16.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:16.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:17.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:17.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:17.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:17.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:18.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:18.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:18.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:18.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:18.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:18.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:19.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:19.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:19.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:19.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:20.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:20.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:20.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:20.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:21.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:21.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:21.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:21.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:22.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:22.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:22.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:22.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:23.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:23.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:23.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:23.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:23.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:23.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:24.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:24.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:24.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:24.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:25.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:25.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:25.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:25.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:26.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:26.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:26.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:26.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:27.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:27.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:28.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:28.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:28.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:28.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:28.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:28.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:29.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:29.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:29.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:29.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:30.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:30.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:31.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:31.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:31.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:31.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:32.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:32.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:32.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:32.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:33.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:33.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:33.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:33.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:33.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:33.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:34.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:34.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:34.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:34.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:35.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:35.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:35.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:35.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:36.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:36.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:36.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:36.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:37.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:37.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:37.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:37.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:38.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:38.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:38.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:38.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:38.969 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:38.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:39.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:39.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:39.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:39.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:40.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:40.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:40.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:40.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:41.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:41.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:41.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:41.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:42.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:42.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:42.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:42.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:43.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:43.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:43.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:43.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:43.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:43.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:44.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:44.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:44.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:44.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:45.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:45.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:45.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:45.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:46.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:46.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:46.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:46.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:47.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:47.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:47.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:47.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:48.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:48.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:48.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:48.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:48.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:48.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:49.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:49.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:49.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:49.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:50.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:50.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:50.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:50.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:51.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:51.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:51.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:51.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:52.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:52.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:52.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:52.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:53.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:53.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:53.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:53.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:53.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:53.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:54.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:54.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:54.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:54.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:55.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:55.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:55.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:55.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:56.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:56.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:56.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:56.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:57.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:57.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:57.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:57.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:58.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:58.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:58.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:58.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:09:58.972 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:09:58.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:09:59.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:59.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:09:59.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:09:59.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:00.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:00.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:00.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:01.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:01.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:01.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:01.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:02.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:02.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:02.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:02.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:03.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:03.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:03.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:03.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:03.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:03.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:04.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:04.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:04.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:04.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:05.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:05.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:05.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:05.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:06.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:06.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:06.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:06.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:07.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:07.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:07.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:07.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:08.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:08.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:08.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:08.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:08.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:08.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:09.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:09.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:09.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:09.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:10.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:10.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:10.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:10.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:11.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:11.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:11.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:11.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:12.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:12.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:12.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:12.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:13.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:13.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:13.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:13.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:13.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:13.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:14.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:14.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:14.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:14.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:15.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:15.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:16.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:16.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:16.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:16.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:17.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:17.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:17.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:17.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:18.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:18.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:18.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:18.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:18.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:18.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:19.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:19.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:19.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:19.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:20.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:20.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:20.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:20.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:21.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:21.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:21.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:21.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:22.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:22.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:22.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:22.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:23.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:23.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:23.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:23.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:23.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:23.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:24.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:24.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:24.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:24.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:25.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:25.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:25.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:25.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:26.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:26.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:26.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:26.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:27.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:27.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:27.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:27.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:28.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:28.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:28.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:28.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:28.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:28.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:29.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:29.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:29.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:29.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:30.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:30.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:30.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:30.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:31.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:31.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:31.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:31.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:32.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:32.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:32.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:32.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:33.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:33.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:33.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:33.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:33.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:33.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:34.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:34.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:34.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:34.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:35.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:35.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:35.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:35.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:36.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:36.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:36.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:36.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:37.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:37.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:37.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:37.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:38.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:38.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:38.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:38.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:38.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:38.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:39.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:39.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:39.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:39.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:40.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:40.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:40.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:40.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:41.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:41.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:41.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:42.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:42.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:42.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:42.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:43.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:43.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:43.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:43.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:43.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:43.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:44.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:44.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:44.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:44.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:45.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:45.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:45.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:45.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:46.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:46.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:46.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:46.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:47.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:47.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:47.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:47.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:48.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:48.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:48.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:48.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:48.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:48.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:49.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:49.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:49.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:49.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:50.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:50.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:50.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:50.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:51.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:51.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:51.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:51.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:52.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:52.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:52.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:52.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:53.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:53.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:53.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:53.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:53.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:53.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:54.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:54.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:54.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:54.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:55.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:55.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:55.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:55.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:56.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:56.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:57.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:57.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:57.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:57.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:58.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:58.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:58.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:58.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:10:58.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:10:58.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:10:59.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:59.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:10:59.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:10:59.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:00.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:00.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:00.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:00.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:01.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:01.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:01.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:01.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:02.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:02.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:02.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:02.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:03.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:03.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:03.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:03.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:03.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:03.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:04.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:04.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:04.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:04.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:05.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:05.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:05.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:05.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:06.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:06.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:06.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:06.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:07.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:07.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:07.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:07.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:08.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:08.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:08.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:08.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:08.984 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:08.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:09.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:09.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:09.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:09.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:10.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:10.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:10.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:10.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:11.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:11.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:11.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:11.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:12.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:12.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:12.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:12.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:13.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:13.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:13.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:13.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:13.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:13.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:14.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:14.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:14.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:14.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:15.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:15.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:15.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:15.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:16.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:16.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:16.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:16.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:17.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:17.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:17.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:17.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:18.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:18.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:18.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:18.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:18.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:18.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:19.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:19.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:19.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:19.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:20.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:20.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:20.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:20.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:21.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:21.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:21.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:21.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:22.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:22.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:23.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:23.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:23.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:23.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:23.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:23.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:24.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:24.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:24.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:24.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:25.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:25.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:25.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:25.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:26.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:26.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:26.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:26.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:27.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:27.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:27.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:27.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:28.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:28.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:28.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:28.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:28.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:28.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:29.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:29.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:29.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:29.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:30.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:30.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:30.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:30.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:31.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:31.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:31.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:31.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:32.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:32.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:32.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:32.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:33.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:33.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:33.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:33.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:34.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:34.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:34.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:34.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:35.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:35.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:35.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:35.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:36.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:36.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:36.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:36.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:37.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:37.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:37.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:37.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:38.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:38.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:38.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:38.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:38.989 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:38.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:39.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:39.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:39.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:39.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:40.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:40.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:40.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:40.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:41.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:41.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:41.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:41.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:42.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:42.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:43.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:43.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:43.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:43.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:43.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:43.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:44.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:44.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:44.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:44.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:45.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:45.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:45.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:45.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:46.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:46.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:46.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:46.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:47.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:47.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:47.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:47.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:48.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:48.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:48.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:48.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:48.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:48.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:49.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:49.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:49.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:49.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:50.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:50.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:50.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:50.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:51.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:51.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:51.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:51.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:52.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:52.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:52.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:52.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:53.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:53.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:53.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:53.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:53.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:53.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:54.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:54.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:54.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:54.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:55.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:55.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:55.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:55.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:56.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:56.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:56.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:56.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:57.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:57.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:57.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:57.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:58.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:58.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:58.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:58.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:11:58.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:11:58.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:11:59.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:59.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:11:59.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:11:59.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:00.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:00.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:00.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:00.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:01.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:01.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:01.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:01.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:02.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:02.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:02.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:02.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:03.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:03.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:03.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:03.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:03.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:03.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:04.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:04.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:04.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:04.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:05.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:05.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:05.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:05.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:06.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:06.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:06.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:06.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:07.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:07.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:07.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:07.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:08.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:08.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:08.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:08.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:08.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:08.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:09.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:09.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:09.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:09.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:10.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:10.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:10.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:10.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:11.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:11.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:11.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:11.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:12.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:12.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:12.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:12.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:13.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:13.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:13.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:13.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:13.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:13.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:14.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:14.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:14.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:14.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:15.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:15.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:15.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:15.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:16.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:16.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:16.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:16.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:17.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:17.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:17.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:17.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:18.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:18.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:18.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:18.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:18.996 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:18.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:19.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:19.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:19.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:19.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:20.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:20.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:20.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:20.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:21.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:21.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:21.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:21.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:22.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:22.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:22.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:22.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:23.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:23.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:23.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:23.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:23.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:23.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:24.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:24.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:24.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:24.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:25.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:25.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:25.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:25.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:26.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:26.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:26.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:26.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:27.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:27.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:27.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:27.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:28.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:28.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:28.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:28.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:28.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:28.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:29.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:29.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:29.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:29.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:30.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:30.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:30.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:30.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:31.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:31.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:31.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:31.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:32.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:32.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:32.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:32.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:33.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:33.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:33.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:33.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:33.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:33.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:34.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:34.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:34.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:34.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:35.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:35.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:35.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:35.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:36.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:36.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:36.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:36.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:37.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:37.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:37.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:37.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:38.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:38.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:38.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:38.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:38.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:38.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:39.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:39.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:39.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:39.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:40.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:40.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:40.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:40.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:41.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:41.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:41.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:41.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:42.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:42.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:42.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:42.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:43.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:43.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:43.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:43.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:44.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:43.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:44.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:44.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:44.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:44.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:45.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:45.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:45.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:45.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:46.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:46.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:46.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:46.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:47.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:47.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:47.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:47.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:48.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:48.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:48.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:48.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:49.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:48.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:49.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:49.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:49.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:49.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:50.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:50.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:50.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:50.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:51.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:51.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:51.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:51.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:52.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:52.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:52.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:52.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:53.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:53.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:53.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:53.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:54.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:53.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:54.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:54.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:54.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:54.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:55.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:55.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:55.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:55.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:56.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:56.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:56.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:56.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:57.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:57.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:57.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:57.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:58.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:58.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:58.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:58.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:12:59.003 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:12:58.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:12:59.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:59.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:12:59.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:12:59.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:00.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:00.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:00.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:00.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:01.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:01.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:01.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:01.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:02.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:02.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:02.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:02.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:03.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:03.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:03.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:03.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:04.003 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:03.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:04.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:04.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:04.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:04.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:05.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:05.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:06.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:06.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:06.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:06.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:07.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:07.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:07.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:07.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:08.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:08.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:08.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:08.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:09.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:08.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:09.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:09.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:09.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:09.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:10.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:10.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:10.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:11.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:11.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:11.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:11.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:12.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:12.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:12.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:12.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:13.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:13.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:13.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:13.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:14.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:14.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:14.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:14.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:14.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:14.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:15.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:15.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:15.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:15.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:16.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:16.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:16.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:16.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:17.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:17.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:17.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:17.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:18.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:18.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:18.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:18.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:19.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:19.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:19.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:19.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:19.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:19.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:20.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:20.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:20.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:20.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:21.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:21.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:21.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:21.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:22.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:22.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:22.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:22.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:23.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:23.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:23.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:23.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:24.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:24.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:24.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:24.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:24.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:24.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:25.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:25.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:25.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:25.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:26.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:26.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:26.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:26.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:27.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:27.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:27.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:27.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:28.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:28.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:28.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:28.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:29.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:29.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:29.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:29.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:29.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:29.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:30.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:30.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:30.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:30.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:31.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:31.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:31.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:31.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:32.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:32.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:32.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:32.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:33.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:33.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:33.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:33.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:34.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:34.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:34.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:34.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:34.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:34.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:35.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:35.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:35.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:35.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:36.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:36.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:36.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:36.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:37.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:37.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:37.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:37.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:38.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:38.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:38.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:39.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:39.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:39.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:39.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:39.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:39.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:40.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:40.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:40.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:40.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:41.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:41.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:41.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:41.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:42.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:42.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:42.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:42.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:43.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:43.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:43.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:43.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:44.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:44.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:44.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:44.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:44.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:44.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:45.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:45.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:45.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:45.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:46.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:46.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:46.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:46.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:47.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:47.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:47.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:47.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:48.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:48.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:48.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:48.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:49.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:49.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:49.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:49.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:49.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:49.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:50.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:50.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:50.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:50.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:51.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:51.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:51.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:51.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:52.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:52.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:52.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:52.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:53.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:53.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:53.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:53.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:54.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:54.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:54.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:54.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:54.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:54.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:55.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:55.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:55.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:55.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:56.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:56.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:56.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:56.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:57.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:57.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:57.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:57.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:58.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:58.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:58.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:58.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:13:59.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:13:59.003+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:13:59.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:59.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:13:59.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:13:59.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:00.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:00.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:00.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:00.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:01.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:01.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:01.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:01.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:02.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:02.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:02.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:02.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:03.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:03.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:03.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:03.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:04.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:04.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:04.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:04.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:04.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:04.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:05.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:05.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:05.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:05.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:06.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:06.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:06.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:06.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:07.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:07.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:07.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:07.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:08.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:08.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:08.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:08.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:09.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:09.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:09.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:09.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:09.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:09.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:10.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:10.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:10.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:10.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:11.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:11.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:11.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:11.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:12.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:12.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:12.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:12.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:13.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:13.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:13.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:13.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:14.015 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:14.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:14.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:14.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:14.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:14.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:15.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:15.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:15.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:15.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:16.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:16.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:16.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:16.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:17.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:17.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:17.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:17.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:18.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:18.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:18.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:18.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:19.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:19.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:19.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:19.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:19.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:19.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:20.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:20.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:20.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:20.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:21.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:21.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:21.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:21.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:22.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:22.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:22.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:23.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:23.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:23.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:23.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:24.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:24.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:24.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:24.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:24.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:24.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:25.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:25.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:25.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:25.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:26.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:26.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:26.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:26.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:27.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:27.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:27.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:27.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:28.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:28.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:28.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:28.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:29.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:29.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:29.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:29.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:29.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:29.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:30.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:30.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:30.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:30.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:31.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:31.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:31.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:31.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:32.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:32.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:32.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:32.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:33.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:33.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:33.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:33.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:34.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:34.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:34.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:34.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:34.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:34.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:35.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:35.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:35.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:35.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:36.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:36.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:36.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:36.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:37.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:37.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:37.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:37.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:38.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:38.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:38.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:38.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:39.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:39.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:39.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:39.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:39.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:39.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:40.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:40.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:40.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:40.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:41.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:41.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:41.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:41.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:42.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:42.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:42.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:42.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:43.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:43.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:43.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:43.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:44.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:44.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:44.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:44.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:44.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:44.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:45.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:45.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:45.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:45.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:46.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:46.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:46.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:46.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:47.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:47.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:47.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:47.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:48.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:48.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:48.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:48.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:49.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:49.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:49.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:49.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:49.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:49.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:50.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:50.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:50.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:50.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:51.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:51.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:51.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:51.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:52.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:52.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:52.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:52.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:53.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:53.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:53.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:53.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:54.022 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:54.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:54.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:54.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:54.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:54.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:55.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:55.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:55.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:55.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:56.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:56.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:56.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:56.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:57.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:57.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:57.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:57.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:58.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:58.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:58.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:58.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:14:59.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:14:59.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:14:59.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:59.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:14:59.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:14:59.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:00.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:00.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:00.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:00.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:01.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:01.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:01.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:01.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:02.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:02.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:02.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:02.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:03.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:03.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:03.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:03.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:04.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:04.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:04.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:04.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:04.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:04.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:05.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:05.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:05.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:05.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:06.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:06.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:06.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:06.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:07.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:07.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:07.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:07.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:08.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:08.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:08.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:08.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:09.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:09.013+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:09.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:09.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:09.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:09.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:10.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:10.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:10.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:10.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:11.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:11.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:11.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:11.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:12.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:12.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:12.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:12.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:13.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:13.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:13.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:13.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:14.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:14.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:14.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:14.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:14.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:14.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:15.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:15.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:15.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:15.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:16.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:16.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:16.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:16.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:17.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:17.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:17.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:17.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:18.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:18.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:18.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:18.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:19.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:19.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:19.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:19.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:19.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:19.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:20.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:20.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:20.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:20.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:21.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:21.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:21.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:21.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:22.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:22.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:22.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:22.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:23.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:23.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:23.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:23.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:24.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:24.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:24.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:24.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:24.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:24.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:25.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:25.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:25.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:25.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:26.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:26.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:26.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:26.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:27.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:27.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:27.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:27.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:28.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:28.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:28.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:28.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:29.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:29.021+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:29.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:29.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:29.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:29.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:30.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:30.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:30.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:30.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:31.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:31.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:31.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:31.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:32.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:32.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:32.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:32.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:33.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:33.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:33.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:33.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:34.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:34.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:34.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:34.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:34.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:34.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:35.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:35.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:35.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:35.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:36.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:36.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:36.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:37.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:37.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:37.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:37.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:38.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:38.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:38.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:38.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:39.029 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:39.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:39.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:39.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:39.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:39.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:40.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:40.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:40.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:40.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:41.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:41.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:41.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:41.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:42.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:42.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:42.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:42.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:43.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:43.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:43.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:43.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:44.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:44.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:44.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:44.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:44.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:44.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:45.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:45.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:45.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:45.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:46.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:46.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:46.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:46.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:47.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:47.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:47.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:47.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:48.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:48.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:48.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:48.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:49.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:49.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:49.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:49.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:49.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:49.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:50.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:50.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:50.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:50.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:51.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:51.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:51.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:51.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:52.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:52.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:52.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:52.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:53.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:53.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:53.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:53.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:54.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:54.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:54.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:54.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:54.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:54.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:55.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:55.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:55.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:55.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:56.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:56.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:56.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:56.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:57.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:57.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:57.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:57.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:58.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:58.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:58.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:58.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:15:59.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:15:59.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:15:59.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:59.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:15:59.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:15:59.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:00.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:00.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:00.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:00.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:01.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:01.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:01.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:01.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:02.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:02.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:02.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:02.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:03.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:03.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:03.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:03.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:04.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:04.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:04.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:04.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:04.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:04.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:05.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:05.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:05.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:05.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:06.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:06.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:06.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:06.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:07.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:07.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:07.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:07.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:08.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:08.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:08.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:08.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:09.035 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:09.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:09.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:09.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:09.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:09.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:10.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:10.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:10.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:10.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:11.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:11.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:11.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:11.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:12.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:12.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:12.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:12.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:13.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:13.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:13.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:13.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:14.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:14.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:14.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:14.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:14.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:14.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:15.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:15.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:15.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:15.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:16.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:16.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:16.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:16.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:17.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:17.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:17.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:17.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:18.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:18.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:18.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:18.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:19.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:19.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:19.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:19.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:19.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:19.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:20.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:20.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:20.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:20.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:21.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:21.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:21.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:21.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:22.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:22.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:22.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:22.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:23.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:23.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:23.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:23.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:24.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:24.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:24.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:24.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:24.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:24.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:25.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:25.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:25.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:25.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:26.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:26.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:26.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:26.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:27.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:27.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:27.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:27.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:28.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:28.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:28.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:28.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:29.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:29.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:29.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:29.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:29.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:29.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:30.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:30.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:30.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:30.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:31.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:31.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:31.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:31.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:32.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:32.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:32.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:32.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:33.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:33.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:33.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:33.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:34.039 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:34.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:34.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:34.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:34.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:34.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:35.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:35.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:35.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:35.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:36.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:36.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:36.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:36.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:37.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:37.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:37.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:37.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:38.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:38.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:38.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:38.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:39.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:39.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:39.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:39.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:39.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:39.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:40.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:40.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:40.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:40.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:41.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:41.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:41.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:41.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:42.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:42.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:42.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:42.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:43.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:43.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:44.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:44.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:44.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:44.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:44.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:44.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:45.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:45.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:45.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:45.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:46.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:46.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:46.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:46.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:47.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:47.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:47.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:47.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:48.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:48.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:48.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:48.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:49.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:49.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:49.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:49.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:49.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:49.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:50.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:50.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:50.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:50.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:51.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:51.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:51.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:51.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:52.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:52.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:52.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:52.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:53.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:53.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:53.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:53.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:54.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:54.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:54.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:54.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:54.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:54.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:55.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:55.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:55.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:55.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:56.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:56.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:56.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:56.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:57.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:57.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:58.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:58.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:58.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:58.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:16:59.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:16:59.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:16:59.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:59.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:16:59.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:16:59.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:00.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:00.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:00.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:00.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:01.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:01.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:01.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:01.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:02.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:02.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:02.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:02.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:03.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:03.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:03.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:03.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:04.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:04.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:04.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:04.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:05.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:05.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:05.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:05.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:06.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:06.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:06.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:06.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:07.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:07.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:07.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:07.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:08.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:08.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:08.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:08.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:09.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:09.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:09.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:09.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:09.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:09.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:10.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:10.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:10.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:10.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:11.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:11.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:11.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:11.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:12.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:12.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:12.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:12.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:13.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:13.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:13.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:13.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:14.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:14.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:14.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:14.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:14.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:14.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:15.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:15.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:15.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:15.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:16.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:16.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:16.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:16.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:17.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:17.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:17.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:17.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:18.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:18.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:18.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:18.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:19.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:19.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:19.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:19.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:19.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:19.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:20.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:20.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:20.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:20.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:21.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:21.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:21.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:21.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:22.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:22.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:22.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:22.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:23.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:23.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:23.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:23.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:24.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:24.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:24.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:24.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:24.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:24.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:25.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:25.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:25.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:25.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:26.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:26.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:26.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:26.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:26.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:26.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:26.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:26.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:27.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:27.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:28.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:28.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:28.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:28.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:29.048 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:29.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:29.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:29.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:29.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:29.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:30.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:30.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:30.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:30.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:32.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:32.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:32.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:32.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:32.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:32.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:32.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:32.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:33.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:33.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:33.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:33.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:34.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:34.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:34.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:34.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:34.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:34.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:36.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:36.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:36.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:36.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:36.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:36.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:36.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:36.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:37.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:37.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:37.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:37.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:38.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:38.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:38.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:38.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:39.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:39.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:39.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:39.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:39.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:39.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:40.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:40.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:40.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:40.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:41.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:41.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:41.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:41.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:42.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:42.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:42.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:42.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:43.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:43.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:43.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:43.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:44.051 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:44.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:44.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:44.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:45.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:45.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:45.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:45.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:46.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:46.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:46.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:46.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:47.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:47.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:47.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:47.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:48.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:48.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:48.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:48.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:49.051 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:49.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:49.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:49.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:49.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:49.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:50.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:50.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:50.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:50.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:51.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:51.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:51.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:51.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:52.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:52.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:52.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:52.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:53.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:53.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:53.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:53.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:54.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:54.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:54.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:54.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:54.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:54.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:55.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:55.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:55.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:55.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:56.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:56.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:56.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:56.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:57.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:57.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:57.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:57.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:58.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:58.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:58.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:58.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:17:59.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:17:59.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:17:59.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:59.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:17:59.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:17:59.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:00.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:00.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:00.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:00.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:01.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:01.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:01.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:01.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:02.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:02.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:02.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:02.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:03.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:03.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:04.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:04.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:04.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:04.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:05.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:05.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:05.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:05.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:06.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:06.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:06.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:06.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:07.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:07.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:07.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:07.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:08.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:08.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:08.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:08.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:09.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:09.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:09.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:09.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:09.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:09.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:10.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:10.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:10.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:10.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:11.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:11.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:11.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:11.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:12.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:12.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:12.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:12.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:13.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:13.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:14.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:14.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:14.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:14.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:14.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:14.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:15.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:15.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:15.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:15.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:16.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:16.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:17.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:17.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:17.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:17.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:18.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:18.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:18.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:18.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:19.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:19.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:19.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:19.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:19.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:19.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:20.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:20.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:20.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:20.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:21.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:21.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:21.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:21.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:22.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:22.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:22.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:22.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:23.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:23.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:07:58.526942+0000 (oldest deadline 2019-10-17T23:08:18.526942+0000) 2019-10-17T23:18:23.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:23.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:24.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:24.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:24.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:24.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:25.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:25.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:26.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:26.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:27.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:27.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:28.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:28.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:29.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:29.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:29.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:29.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:30.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:30.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:31.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:31.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:32.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:32.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:33.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:33.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:34.059 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:34.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:34.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:34.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:35.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:35.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:36.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:36.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:37.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:37.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:38.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:38.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:39.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:39.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:39.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:39.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:40.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:41.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:41.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:42.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:42.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:43.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:43.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:44.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:44.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:44.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:44.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:45.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:45.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:46.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:46.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:46.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:46.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:47.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:47.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:47.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:47.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:48.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:48.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:48.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:48.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:49.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:49.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:49.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:49.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:49.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:49.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:50.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:50.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:50.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:50.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:51.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:51.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:51.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:51.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:52.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:52.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:52.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:52.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:53.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:53.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:53.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:53.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:54.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:54.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:54.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:54.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:54.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:54.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:55.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:55.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:55.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:55.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:56.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:56.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:56.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:56.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:57.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:57.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:57.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:57.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:58.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:58.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:58.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:58.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:18:59.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:18:59.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:18:59.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:59.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:18:59.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:18:59.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:00.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:00.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:00.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:00.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:01.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:01.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:01.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:01.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:02.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:02.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:02.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:02.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:03.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:03.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:03.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:03.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:04.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:04.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:04.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:04.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:04.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:04.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:05.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:05.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:05.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:05.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:06.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:06.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:06.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:06.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:07.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:07.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:07.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:07.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:08.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:08.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:08.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:08.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:09.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:09.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:09.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:09.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:10.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:10.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:10.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:10.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:11.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:11.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:11.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:11.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:12.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:12.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:12.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:12.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:13.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:13.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:13.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:13.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:14.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:14.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:14.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:14.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:14.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:14.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:15.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:15.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:15.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:15.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:16.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:16.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:16.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:16.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:17.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:17.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:17.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:17.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:18.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:18.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:18.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:18.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:19.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:19.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:19.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:19.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:19.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:19.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:20.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:20.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:20.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:20.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:21.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:21.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:21.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:21.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:22.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:22.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:22.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:22.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:23.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:23.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:23.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:23.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:24.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:24.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:24.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:24.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:24.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:24.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:25.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:25.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:25.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:25.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:26.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:26.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:26.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:26.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:27.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:27.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:27.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:27.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:28.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:28.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:28.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:28.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:29.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:29.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:29.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:29.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:29.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:29.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:30.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:30.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:30.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:30.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:31.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:31.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:31.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:31.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:32.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:32.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:32.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:32.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:33.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:33.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:33.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:33.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:34.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:34.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:34.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:34.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:34.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:34.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:35.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:35.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:35.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:35.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:36.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:36.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:36.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:36.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:37.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:37.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:37.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:37.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:38.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:38.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:38.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:38.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:39.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:39.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:39.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:39.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:39.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:39.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:40.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:40.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:40.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:40.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:41.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:41.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:41.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:41.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:42.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:42.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:42.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:42.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:43.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:43.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:43.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:43.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:44.071 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:44.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:44.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:44.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:44.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:44.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:45.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:45.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:45.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:45.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:46.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:46.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:46.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:46.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:47.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:47.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:47.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:47.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:48.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:48.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:48.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:48.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:49.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:49.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:49.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:49.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:49.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:49.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:50.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:50.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:50.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:50.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:51.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:51.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:51.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:51.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:52.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:52.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:52.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:52.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:53.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:53.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:53.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:53.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:54.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:54.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:54.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:54.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:55.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:55.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:55.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:55.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:56.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:56.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:56.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:56.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:57.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:57.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:57.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:57.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:58.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:58.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:58.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:58.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:19:59.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:19:59.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:19:59.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:59.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:19:59.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:19:59.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:00.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:00.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:00.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:01.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:01.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:01.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:01.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:02.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:02.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:03.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:03.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:03.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:03.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:04.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:04.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:04.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:04.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:04.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:05.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:05.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:05.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:05.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:06.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:06.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:06.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:06.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:07.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:07.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:07.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:07.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:08.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:08.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:08.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:08.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:09.075 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:09.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:09.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:09.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:09.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:09.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:10.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:10.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:10.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:10.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:11.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:11.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:12.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:12.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:12.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:12.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:13.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:13.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:13.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:13.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:14.076 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:14.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:14.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:14.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:14.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:14.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:15.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:15.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:15.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:15.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:16.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:16.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:16.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:16.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:17.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:17.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:17.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:17.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:18.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:18.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:18.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:18.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:19.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:19.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:19.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:19.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:19.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:19.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:20.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:20.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:20.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:20.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:21.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:21.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:21.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:21.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:22.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:22.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:22.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:22.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:23.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:23.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:23.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:23.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:24.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:24.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:24.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:24.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:24.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:24.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:25.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:25.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:25.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:25.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:26.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:26.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:26.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:26.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:27.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:27.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:27.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:27.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:28.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:28.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:28.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:28.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:29.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:29.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:29.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:29.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:29.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:29.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:30.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:30.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:30.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:30.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:31.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:31.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:31.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:31.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:32.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:32.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:32.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:32.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:33.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:33.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:33.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:33.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:34.080 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:34.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:34.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:34.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:34.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:34.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:35.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:35.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:35.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:35.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:36.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:36.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:36.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:36.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:37.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:37.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:37.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:37.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:38.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:38.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:38.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:38.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:39.080 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:39.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:39.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:39.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:39.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:39.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:40.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:40.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:40.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:40.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:41.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:41.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:41.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:41.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:42.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:42.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:42.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:42.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:43.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:43.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:43.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:43.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:44.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:44.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:44.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:44.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:44.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:44.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:45.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:45.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:45.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:45.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:46.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:46.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:46.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:46.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:47.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:47.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:47.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:47.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:48.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:48.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:48.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:48.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:49.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:49.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:49.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:49.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:49.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:49.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:50.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:50.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:50.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:50.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:51.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:51.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:51.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:51.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:52.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:52.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:52.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:52.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:53.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:53.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:53.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:53.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:54.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:54.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:54.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:54.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:54.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:54.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:55.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:55.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:55.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:55.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:56.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:56.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:56.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:56.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:57.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:57.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:57.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:57.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:58.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:58.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:58.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:58.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:20:59.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:20:59.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:20:59.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:59.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:20:59.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:20:59.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:00.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:00.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:00.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:00.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:01.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:01.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:01.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:01.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:02.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:02.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:02.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:02.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:03.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:03.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:03.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:03.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:04.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:04.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:04.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:04.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:04.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:04.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:05.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:05.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:05.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:05.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:06.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:06.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:06.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:06.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:07.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:07.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:07.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:07.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:08.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:08.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:08.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:08.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:09.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:09.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:09.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:09.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:09.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:09.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:10.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:10.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:10.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:10.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:11.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:11.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:11.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:11.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:12.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:12.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:12.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:12.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:13.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:13.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:13.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:13.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:14.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:14.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:14.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:14.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:14.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:14.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:15.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:15.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:15.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:15.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:16.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:16.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:16.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:16.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:17.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:17.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:17.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:17.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:18.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:18.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:18.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:18.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:19.087 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:19.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:19.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:19.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:19.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:19.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:20.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:20.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:20.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:20.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:21.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:21.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:21.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:21.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:22.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:22.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:22.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:22.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:23.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:23.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:23.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:23.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:24.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:24.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:24.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:24.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:24.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:24.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:25.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:25.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:25.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:25.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:26.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:26.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:26.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:26.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:27.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:27.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:27.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:27.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:28.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:28.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:28.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:28.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:29.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:29.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:29.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:29.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:29.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:29.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:30.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:30.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:30.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:30.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:31.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:31.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:31.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:31.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:32.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:32.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:32.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:32.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:33.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:33.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:33.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:33.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:34.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:34.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:34.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:34.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:34.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:34.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:35.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:35.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:35.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:35.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:36.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:36.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:36.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:36.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:37.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:37.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:37.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:37.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:38.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:38.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:39.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:39.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:39.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:39.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:39.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:39.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:40.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:40.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:40.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:40.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:41.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:41.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:41.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:41.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:42.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:42.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:42.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:42.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:43.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:43.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:43.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:43.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:44.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:44.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:44.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:44.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:44.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:44.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:45.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:45.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:45.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:45.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:46.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:46.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:46.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:46.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:47.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:47.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:47.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:47.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:48.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:48.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:48.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:48.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:49.092 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:49.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:49.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:49.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:49.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:49.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:50.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:50.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:50.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:50.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:51.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:51.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:51.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:51.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:52.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:52.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:52.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:52.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:53.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:53.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:53.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:53.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:54.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:54.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:54.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:54.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:54.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:54.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:55.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:55.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:55.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:55.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:56.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:56.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:56.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:56.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:57.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:57.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:57.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:57.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:58.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:58.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:58.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:58.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:21:59.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:21:59.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:21:59.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:59.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:21:59.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:21:59.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:00.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:00.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:00.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:00.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:01.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:01.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:01.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:01.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:02.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:02.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:02.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:02.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:03.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:03.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:03.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:03.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:04.095 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:04.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:04.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:04.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:04.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:04.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:05.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:05.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:05.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:05.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:06.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:06.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:06.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:06.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:07.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:07.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:07.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:07.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:08.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:08.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:08.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:08.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:09.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:09.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:09.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:09.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:09.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:09.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:10.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:10.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:10.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:10.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:11.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:11.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:11.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:11.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:12.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:12.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:12.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:12.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:13.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:13.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:13.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:13.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:14.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:14.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:14.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:14.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:14.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:14.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:15.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:15.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:16.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:16.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:16.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:16.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:17.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:17.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:17.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:17.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:18.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:18.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:19.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:19.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:19.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:19.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:19.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:19.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:20.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:20.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:20.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:20.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:21.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:21.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:21.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:21.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:22.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:22.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:22.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:22.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:23.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:23.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:23.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:23.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:24.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:24.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:24.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:24.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:25.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:25.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:25.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:26.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:26.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:26.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:26.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:27.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:27.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:27.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:27.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:28.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:28.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:28.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:28.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:29.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:29.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:29.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:29.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:29.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:29.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:30.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:30.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:30.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:30.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:31.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:31.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:31.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:31.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:32.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:32.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:32.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:32.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:33.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:33.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:33.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:33.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:34.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:34.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:34.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:34.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:34.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:34.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:35.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:35.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:35.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:35.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:36.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:36.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:36.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:36.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:37.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:37.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:37.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:37.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:38.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:38.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:38.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:38.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:39.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:39.097+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:39.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:39.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:39.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:39.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:40.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:40.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:40.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:40.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:41.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:41.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:41.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:41.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:42.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:42.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:43.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:43.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:43.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:43.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:44.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:44.097+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:44.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:44.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:44.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:44.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:45.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:45.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:46.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:46.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:46.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:46.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:47.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:47.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:47.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:47.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:48.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:48.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:48.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:48.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:49.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:49.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:49.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:49.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:49.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:49.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:50.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:50.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:50.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:50.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:51.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:51.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:51.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:51.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:52.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:52.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:52.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:52.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:53.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:53.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:53.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:53.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:54.103 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:54.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:54.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:54.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:54.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:54.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:55.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:55.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:55.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:55.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:56.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:56.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:56.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:56.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:57.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:57.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:57.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:57.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:58.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:58.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:58.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:58.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:22:59.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:22:59.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:22:59.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:59.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:22:59.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:22:59.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:00.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:00.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:00.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:00.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:01.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:01.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:01.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:01.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:02.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:02.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:02.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:02.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:03.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:03.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:03.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:03.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:04.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:04.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:04.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:04.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:04.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:04.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:05.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:05.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:05.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:05.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:06.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:06.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:06.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:06.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:07.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:07.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:07.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:07.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:08.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:08.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:08.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:08.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:09.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:09.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:09.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:09.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:09.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:09.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:10.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:10.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:10.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:10.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:11.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:11.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:11.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:11.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:12.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:12.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:12.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:12.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:13.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:13.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:13.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:13.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:14.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:14.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:14.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:14.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:14.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:14.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:15.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:15.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:15.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:15.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:16.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:16.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:16.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:16.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:17.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:17.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:17.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:17.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:18.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:18.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:18.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:18.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:19.108 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:19.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:19.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:19.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:19.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:19.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:20.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:20.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:20.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:20.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:21.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:21.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:21.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:21.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:22.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:22.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:22.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:22.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:23.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:23.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:23.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:23.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:24.108 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:24.099+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:24.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:24.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:24.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:24.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:25.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:25.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:25.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:25.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:26.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:26.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:26.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:26.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:27.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:27.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:27.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:27.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:28.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:28.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:28.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:28.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:29.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:29.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:29.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:29.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:29.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:29.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:30.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:30.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:30.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:30.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:31.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:31.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:31.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:31.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:32.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:32.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:32.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:32.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:33.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:33.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:33.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:33.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:34.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:34.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:34.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:34.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:34.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:34.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:35.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:35.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:35.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:35.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:36.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:36.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:36.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:36.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:37.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:37.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:37.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:37.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:38.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:38.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:38.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:38.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:39.111 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:39.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:39.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:39.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:39.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:39.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:40.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:40.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:40.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:40.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:41.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:41.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:41.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:41.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:42.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:42.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:42.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:42.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:43.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:43.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:43.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:43.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:44.111 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:44.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:44.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:44.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:44.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:44.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:45.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:45.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:45.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:45.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:46.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:46.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:46.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:46.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:47.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:47.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:47.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:47.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:48.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:48.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:48.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:48.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:49.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:49.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:49.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:49.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:49.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:49.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:50.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:50.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:50.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:50.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:51.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:51.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:51.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:51.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:52.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:52.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:52.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:52.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:53.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:53.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:53.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:53.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:54.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:54.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:54.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:54.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:54.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:54.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:55.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:55.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:55.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:55.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:56.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:56.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:56.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:56.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:57.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:57.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:57.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:57.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:58.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:58.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:58.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:58.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:23:59.114 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:23:59.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:23:59.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:59.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:23:59.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:23:59.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:00.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:00.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:00.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:00.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:01.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:01.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:01.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:01.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:02.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:02.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:02.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:02.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:03.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:03.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:03.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:03.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:04.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:04.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:04.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:04.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:04.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:04.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:05.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:05.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:05.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:05.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:06.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:06.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:06.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:06.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:07.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:07.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:07.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:07.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:08.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:08.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:08.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:08.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:09.116 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:09.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:09.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:09.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:09.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:09.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:10.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:10.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:10.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:10.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:11.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:11.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:11.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:11.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:12.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:12.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:12.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:12.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:13.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:13.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:13.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:13.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:14.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:14.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:14.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:14.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:14.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:14.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:15.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:15.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:15.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:15.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:16.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:16.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:16.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:16.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:17.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:17.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:17.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:17.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:18.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:18.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:18.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:18.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:19.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:19.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:19.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:19.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:19.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:19.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:20.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:20.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:20.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:20.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:21.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:21.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:21.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:21.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:22.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:22.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:22.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:22.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:23.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:23.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:23.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:23.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:24.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:24.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:24.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:24.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:24.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:24.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:25.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:25.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:25.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:25.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:26.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:26.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:26.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:26.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:27.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:27.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:27.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:27.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:28.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:28.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:28.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:28.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:29.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:29.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:29.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:29.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:29.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:29.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:30.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:30.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:30.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:30.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:31.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:31.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:31.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:31.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:32.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:32.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:32.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:32.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:33.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:33.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:33.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:33.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:34.120 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:34.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:34.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:34.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:34.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:34.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:35.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:35.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:35.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:35.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:36.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:36.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:36.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:36.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:37.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:37.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:37.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:37.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:38.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:38.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:38.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:38.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:39.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:39.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:39.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:39.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:39.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:39.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:40.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:40.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:40.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:40.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:41.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:41.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:41.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:41.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:42.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:42.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:42.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:42.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:43.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:43.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:43.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:43.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:44.122 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:44.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:44.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:44.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:44.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:44.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:45.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:45.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:45.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:45.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:46.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:46.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:46.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:46.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:47.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:47.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:47.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:47.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:48.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:48.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:48.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:48.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:49.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:49.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:49.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:49.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:49.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:49.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:50.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:50.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:50.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:50.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:51.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:51.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:51.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:51.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:52.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:52.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:52.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:52.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:53.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:53.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:53.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:53.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:54.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:54.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:24:54.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:54.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:54.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:54.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:55.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:55.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:55.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:55.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:56.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:56.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:56.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:56.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:57.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:57.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:57.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:57.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:58.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:58.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:24:58.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:58.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:24:59.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:24:59.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:00.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:59.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:00.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:24:59.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:01.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:00.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:01.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:00.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:01.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:01.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:01.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:01.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:02.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:02.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:02.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:02.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:03.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:03.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:03.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:03.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:04.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:04.121+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:05.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:04.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:05.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:04.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:05.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:05.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:05.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:05.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:06.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:06.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:06.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:06.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:08.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:07.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:08.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:07.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:09.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:09.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:09.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:09.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:09.126 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:09.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:10.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:10.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:10.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:10.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:11.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:11.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:11.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:11.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:12.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:12.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:12.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:12.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:13.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:13.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:13.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:13.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:14.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:14.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:14.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:14.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:14.127 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:14.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:15.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:15.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:15.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:15.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:16.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:16.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:16.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:16.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:17.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:17.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:17.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:17.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:18.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:18.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:18.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:18.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:19.127 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:19.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:19.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:19.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:19.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:19.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:20.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:20.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:20.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:20.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:21.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:21.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:21.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:21.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:22.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:22.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:22.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:22.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:23.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:23.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:23.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:23.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:24.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:24.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:24.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:24.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:24.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:24.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:25.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:25.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:25.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:25.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:26.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:26.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:26.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:26.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:27.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:27.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:27.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:27.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:28.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:28.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:28.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:28.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:29.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:29.124+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:29.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:29.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:29.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:29.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:30.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:30.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:30.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:30.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:31.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:31.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:31.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:31.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:32.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:32.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:32.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:32.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:33.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:33.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:33.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:33.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:34.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:34.124+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:34.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:34.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:34.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:34.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:35.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:35.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:35.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:35.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:36.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:36.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:36.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:36.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:37.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:37.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:37.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:37.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:38.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:38.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:38.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:38.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:39.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:39.124+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:39.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:39.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:39.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:39.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:40.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:40.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:40.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:40.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:41.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:41.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:41.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:41.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:42.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:42.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:42.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:42.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:43.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:43.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:43.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:43.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:44.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:44.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:44.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:44.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:44.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:44.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:45.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:45.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:46.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:46.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:46.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:46.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:47.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:47.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:47.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:47.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:48.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:48.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:48.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:48.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:49.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:49.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:49.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:49.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:49.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:49.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:50.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:50.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:50.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:50.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:51.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:51.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:51.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:51.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:52.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:52.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:52.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:52.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:53.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:53.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:53.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:53.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:54.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:54.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:54.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:54.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:54.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:54.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:55.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:55.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:55.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:55.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:56.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:56.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:56.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:56.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:57.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:57.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:57.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:57.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:58.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:58.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:58.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:58.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:25:59.134 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:25:59.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:25:59.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:59.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:25:59.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:25:59.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:00.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:00.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:01.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:01.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:01.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:01.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:02.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:02.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:02.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:02.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:03.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:03.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:03.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:03.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:04.135 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:04.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:04.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:04.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:04.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:04.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:05.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:05.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:05.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:05.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:06.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:06.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:06.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:06.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:07.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:07.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:07.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:07.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:08.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:08.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:08.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:08.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:09.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:09.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:09.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:09.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:09.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:09.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:10.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:10.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:10.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:10.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:11.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:11.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:11.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:11.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:12.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:12.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:12.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:12.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:13.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:13.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:13.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:13.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:14.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:14.131+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:14.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:14.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:14.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:14.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:15.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:15.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:15.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:15.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:16.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:16.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:16.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:16.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:17.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:17.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:17.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:17.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:18.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:18.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:18.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:18.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:19.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:19.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:19.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:19.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:19.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:19.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:20.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:20.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:20.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:20.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:21.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:21.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:21.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:21.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:22.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:22.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:22.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:22.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:23.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:23.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:23.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:23.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:24.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:24.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:24.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:24.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:24.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:24.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:25.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:25.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:25.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:25.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:26.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:26.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:26.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:26.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:27.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:27.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:27.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:27.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:28.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:28.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:28.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:28.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:29.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:29.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:29.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:29.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:29.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:29.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:30.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:30.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:30.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:30.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:31.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:31.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:31.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:31.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:32.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:32.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:32.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:32.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:33.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:33.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:33.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:33.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:34.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:34.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:34.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:34.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:34.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:34.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:35.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:35.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:35.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:35.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:36.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:36.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:37.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:37.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:37.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:37.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:38.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:38.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:38.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:38.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:39.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:39.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:39.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:39.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:39.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:39.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:40.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:40.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:40.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:40.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:41.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:41.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:41.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:41.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:42.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:42.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:42.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:42.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:43.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:43.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:43.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:43.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:44.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:44.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:44.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:44.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:44.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:44.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:45.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:45.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:45.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:45.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:46.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:46.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:46.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:46.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:47.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:47.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:47.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:47.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:48.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:48.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:48.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:48.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:49.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:49.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:49.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:49.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:49.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:49.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:50.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:50.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:50.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:50.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:51.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:51.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:51.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:51.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:52.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:52.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:52.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:52.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:53.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:53.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:53.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:53.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:54.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:54.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:54.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:54.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:54.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:54.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:55.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:55.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:55.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:55.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:56.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:56.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:56.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:56.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:57.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:57.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:57.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:57.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:58.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:58.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:58.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:58.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:26:59.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:26:59.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:26:59.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:59.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:26:59.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:26:59.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:00.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:00.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:00.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:00.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:01.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:01.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:01.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:01.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:02.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:02.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:02.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:02.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:03.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:03.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:03.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:03.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:04.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:04.137+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:04.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:04.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:04.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:04.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:05.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:05.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:05.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:05.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:06.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:06.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:06.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:07.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:07.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:07.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:07.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:08.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:08.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:08.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:08.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:09.146 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:09.137+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:09.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:09.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:09.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:09.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:10.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:10.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:10.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:10.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:11.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:11.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:11.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:12.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:12.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:12.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:12.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:13.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:13.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:13.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:13.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:14.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:14.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:14.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:14.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:14.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:14.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:15.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:15.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:15.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:15.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:16.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:16.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:16.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:16.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:17.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:17.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:17.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:17.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:18.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:18.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:18.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:18.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:19.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:19.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:19.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:19.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:19.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:19.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:20.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:20.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:20.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:20.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:21.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:21.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:21.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:21.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:22.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:22.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:22.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:22.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:23.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:23.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:23.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:23.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:24.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:24.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:24.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:24.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:24.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:24.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:25.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:25.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:25.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:25.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:26.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:26.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:26.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:26.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:27.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:27.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:27.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:27.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:28.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:28.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:28.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:28.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:29.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:29.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:29.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:29.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:29.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:29.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:30.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:30.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:30.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:30.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:31.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:31.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:31.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:31.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:32.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:32.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:32.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:32.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:33.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:33.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:33.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:33.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:34.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:34.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:34.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:34.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:34.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:34.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:35.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:35.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:35.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:35.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:36.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:36.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:36.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:36.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:37.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:37.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:37.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:37.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:38.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:38.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:38.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:38.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:39.151 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:39.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:39.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:39.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:39.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:39.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:40.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:40.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:40.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:40.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:41.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:41.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:42.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:42.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:42.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:42.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:43.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:43.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:43.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:43.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:44.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:44.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:44.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:44.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:44.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:44.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:45.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:45.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:45.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:45.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:46.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:46.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:46.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:46.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:47.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:47.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:47.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:47.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:48.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:48.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:48.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:48.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:49.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:49.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:49.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:49.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:49.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:49.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:50.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:50.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:50.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:50.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:51.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:51.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:51.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:51.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:52.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:52.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:52.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:52.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:53.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:53.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:53.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:53.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:54.154 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:54.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:54.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:54.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:54.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:54.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:55.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:55.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:55.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:55.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:56.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:56.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:56.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:56.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:57.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:57.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:57.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:57.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:58.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:58.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:58.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:58.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:27:59.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:27:59.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:27:59.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:59.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:27:59.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:27:59.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:00.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:00.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:00.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:00.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:01.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:01.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:01.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:01.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:02.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:02.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:02.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:02.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:03.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:03.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:03.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:03.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:04.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:04.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:04.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:04.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:04.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:04.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:05.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:05.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:05.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:05.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:06.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:06.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:06.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:06.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:07.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:07.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:07.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:07.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:08.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:08.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:08.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:08.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:09.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:09.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:09.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:09.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:09.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:09.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:10.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:10.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:10.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:10.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:11.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:11.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:11.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:11.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:12.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:12.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:12.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:12.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:13.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:13.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:13.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:13.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:14.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:14.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:14.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:14.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:14.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:14.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:15.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:15.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:15.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:15.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:16.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:16.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:16.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:16.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:17.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:17.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:17.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:17.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:18.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:18.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:18.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:18.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:19.158 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:19.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:19.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:19.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:19.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:19.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:20.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:20.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:20.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:20.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:21.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:21.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:21.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:21.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:22.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:22.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:22.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:22.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:23.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:23.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:23.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:23.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:24.159 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:24.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:24.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:24.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:24.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:24.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:25.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:25.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:25.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:25.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:26.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:26.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:26.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:26.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:27.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:27.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:27.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:27.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:28.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:28.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:28.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:28.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:29.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:29.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:29.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:29.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:29.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:29.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:30.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:30.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:30.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:30.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:31.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:31.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:31.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:31.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:32.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:32.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:32.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:32.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:33.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:33.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:33.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:33.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:34.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:34.155+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:34.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:34.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:34.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:34.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:35.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:35.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:35.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:35.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:36.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:36.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:36.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:36.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:37.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:37.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:37.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:37.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:38.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:38.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:38.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:38.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:39.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:39.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:39.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:39.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:39.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:39.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:40.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:40.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:40.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:40.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:41.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:41.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:41.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:41.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:42.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:42.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:42.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:42.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:43.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:43.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:43.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:43.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:44.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:44.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:44.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:44.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:44.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:44.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:45.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:45.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:45.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:45.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:46.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:46.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:46.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:46.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:47.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:47.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:47.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:47.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:48.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:48.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:48.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:48.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:49.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:49.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:49.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:49.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:49.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:49.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:50.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:50.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:50.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:50.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:51.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:51.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:51.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:51.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:52.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:52.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:52.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:52.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:53.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:53.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:53.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:53.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:54.164 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:54.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:54.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:54.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:54.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:54.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:55.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:55.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:55.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:55.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:56.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:56.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:18:26.257461+0000 (oldest deadline 2019-10-17T23:18:46.257461+0000) 2019-10-17T23:28:56.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:56.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:57.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:57.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:58.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:58.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:28:59.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:28:59.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:28:59.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:28:59.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:00.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:00.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:01.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:01.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:02.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:02.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:03.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:03.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:04.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:04.158+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:04.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:04.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:05.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:05.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:06.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:06.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:07.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:07.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:08.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:08.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:09.167 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:09.158+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:09.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:09.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:10.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:10.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:11.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:11.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:12.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:12.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:13.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:13.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:14.167 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:14.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:14.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:14.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:15.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:15.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:16.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:16.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:17.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:17.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:18.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:18.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:19.168 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:19.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:19.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:19.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:20.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:20.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:20.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:20.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:21.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:21.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:21.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:21.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:22.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:22.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:22.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:22.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:23.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:23.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:23.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:23.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:24.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:24.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:24.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:24.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:24.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:24.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:25.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:25.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:25.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:26.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:26.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:26.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:26.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:27.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:27.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:27.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:27.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:28.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:28.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:28.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:28.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:29.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:29.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:29.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:29.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:29.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:29.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:30.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:30.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:30.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:30.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:31.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:31.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:32.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:32.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:32.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:32.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:33.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:33.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:33.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:33.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:34.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:34.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:34.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:34.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:34.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:34.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:35.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:35.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:35.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:35.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:36.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:36.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:36.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:36.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:37.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:37.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:37.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:37.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:38.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:38.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:38.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:38.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:39.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:39.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:39.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:39.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:39.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:39.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:40.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:40.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:40.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:40.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:41.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:41.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:41.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:41.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:42.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:42.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:42.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:42.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:43.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:43.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:43.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:43.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:44.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:44.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:44.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:44.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:44.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:44.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:45.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:45.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:45.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:45.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:46.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:46.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:46.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:46.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:47.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:47.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:47.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:47.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:48.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:48.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:48.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:48.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:49.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:49.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:49.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:49.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:49.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:49.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:50.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:50.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:50.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:50.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:51.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:51.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:51.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:51.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:52.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:52.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:52.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:52.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:53.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:53.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:53.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:53.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:54.174 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:54.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:54.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:54.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:54.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:54.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:55.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:55.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:55.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:55.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:56.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:56.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:56.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:56.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:57.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:57.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:57.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:57.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:58.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:58.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:58.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:58.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:29:59.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:29:59.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:29:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:59.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:29:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:29:59.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:00.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:00.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:00.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:00.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:01.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:01.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:01.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:01.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:02.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:02.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:02.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:02.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:03.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:03.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:03.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:03.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:04.176 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:04.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:04.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:04.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:04.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:04.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:05.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:05.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:05.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:05.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:06.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:06.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:06.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:06.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:07.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:07.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:07.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:07.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:08.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:08.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:08.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:08.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:09.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:09.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:09.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:09.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:09.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:09.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:10.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:10.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:10.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:10.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:11.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:11.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:11.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:11.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:12.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:12.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:12.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:12.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:13.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:13.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:13.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:13.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:14.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:14.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:14.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:14.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:14.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:14.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:15.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:15.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:15.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:15.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:16.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:16.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:16.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:16.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:17.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:17.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:17.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:17.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:18.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:18.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:18.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:18.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:19.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:19.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:19.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:19.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:19.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:19.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:20.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:20.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:20.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:20.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:21.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:21.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:21.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:21.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:22.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:22.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:22.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:22.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:23.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:23.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:23.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:23.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:24.179 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:24.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:24.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:24.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:24.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:24.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:25.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:25.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:25.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:25.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:26.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:26.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:26.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:26.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:27.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:27.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:27.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:27.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:28.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:28.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:28.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:28.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:29.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:29.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:29.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:29.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:29.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:29.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:30.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:30.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:30.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:30.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:31.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:31.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:31.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:31.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:32.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:32.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:32.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:32.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:33.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:33.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:33.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:33.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:34.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:34.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:34.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:34.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:34.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:34.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:35.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:35.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:35.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:35.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:36.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:36.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:36.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:36.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:37.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:37.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:37.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:37.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:38.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:38.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:38.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:38.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:39.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:39.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:39.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:39.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:39.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:39.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:40.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:40.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:40.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:40.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:41.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:41.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:41.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:41.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:42.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:42.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:42.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:42.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:43.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:43.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:43.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:43.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:44.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:44.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:44.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:44.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:44.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:44.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:45.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:45.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:45.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:45.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:46.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:46.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:46.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:46.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:47.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:47.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:47.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:47.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:48.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:48.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:48.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:48.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:49.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:49.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:49.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:49.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:49.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:49.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:50.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:50.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:50.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:51.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:51.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:51.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:51.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:52.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:52.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:52.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:52.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:53.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:53.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:53.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:54.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:54.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:54.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:54.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:54.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:54.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:55.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:55.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:55.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:55.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:56.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:56.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:56.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:56.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:57.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:57.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:57.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:57.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:58.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:58.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:58.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:58.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:30:59.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:30:59.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:30:59.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:59.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:30:59.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:30:59.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:00.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:00.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:00.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:01.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:01.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:01.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:01.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:02.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:02.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:02.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:02.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:03.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:03.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:03.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:03.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:04.186 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:04.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:04.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:04.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:04.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:04.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:05.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:05.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:05.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:05.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:06.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:06.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:06.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:06.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:07.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:07.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:07.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:07.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:08.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:08.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:08.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:08.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:09.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:09.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:09.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:09.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:09.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:09.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:10.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:10.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:10.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:10.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:11.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:11.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:11.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:11.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:12.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:12.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:12.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:12.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:13.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:13.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:13.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:13.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:14.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:14.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:14.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:14.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:14.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:14.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:15.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:15.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:15.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:15.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:16.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:16.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:16.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:16.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:17.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:17.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:17.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:17.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:18.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:18.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:18.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:18.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:19.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:19.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:19.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:19.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:19.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:19.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:20.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:20.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:20.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:20.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:21.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:21.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:21.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:21.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:22.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:22.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:22.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:22.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:23.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:23.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:23.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:23.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:24.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:24.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:24.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:24.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:24.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:24.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:25.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:25.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:25.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:25.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:26.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:26.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:26.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:26.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:27.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:27.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:27.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:27.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:28.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:28.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:28.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:28.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:29.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:29.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:29.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:29.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:29.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:29.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:30.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:30.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:30.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:30.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:31.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:31.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:31.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:31.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:32.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:32.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:32.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:32.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:33.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:33.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:33.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:33.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:34.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:34.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:34.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:34.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:34.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:34.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:35.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:35.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:35.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:35.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:36.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:36.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:36.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:36.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:37.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:37.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:37.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:37.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:38.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:38.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:38.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:38.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:39.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:39.181+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:39.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:39.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:39.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:39.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:40.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:40.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:40.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:40.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:41.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:41.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:41.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:41.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:42.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:42.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:42.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:42.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:43.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:43.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:43.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:43.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:44.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:44.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:44.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:44.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:44.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:44.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:45.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:45.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:45.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:45.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:46.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:46.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:46.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:46.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:47.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:47.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:47.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:47.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:48.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:48.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:48.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:48.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:49.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:49.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:49.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:49.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:49.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:49.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:50.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:50.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:50.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:50.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:51.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:51.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:51.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:51.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:52.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:52.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:52.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:52.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:53.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:53.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:53.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:53.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:54.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:54.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:54.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:54.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:54.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:54.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:55.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:55.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:55.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:55.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:56.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:56.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:56.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:56.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:57.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:57.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:57.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:57.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:58.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:58.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:58.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:58.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:31:59.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:31:59.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:31:59.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:59.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:31:59.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:31:59.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:00.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:00.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:00.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:00.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:01.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:01.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:01.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:01.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:02.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:02.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:02.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:02.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:03.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:03.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:03.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:03.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:04.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:04.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:04.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:04.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:04.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:04.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:05.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:05.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:05.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:05.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:06.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:06.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:06.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:06.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:07.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:07.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:07.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:07.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:08.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:08.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:08.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:08.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:09.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:09.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:09.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:09.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:09.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:09.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:10.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:10.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:10.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:10.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:11.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:11.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:11.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:11.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:12.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:12.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:12.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:12.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:13.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:13.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:13.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:13.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:14.198 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:14.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:14.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:14.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:14.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:14.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:15.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:15.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:15.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:15.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:16.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:16.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:16.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:16.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:17.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:17.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:17.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:17.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:18.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:18.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:18.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:18.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:19.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:19.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:19.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:19.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:19.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:19.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:20.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:20.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:20.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:20.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:21.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:21.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:21.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:21.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:22.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:22.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:22.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:22.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:23.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:23.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:23.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:23.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:24.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:24.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:24.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:24.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:24.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:24.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:25.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:25.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:25.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:25.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:26.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:26.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:26.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:26.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:27.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:27.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:27.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:27.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:28.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:28.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:28.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:28.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:29.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:29.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:29.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:29.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:29.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:29.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:30.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:30.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:30.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:30.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:31.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:31.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:31.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:31.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:32.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:32.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:32.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:32.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:33.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:33.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:33.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:33.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:34.202 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:34.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:34.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:34.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:34.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:34.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:35.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:35.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:35.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:35.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:36.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:36.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:36.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:36.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:37.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:37.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:37.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:37.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:38.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:38.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:38.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:38.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:39.202 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:39.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:39.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:39.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:39.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:39.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:40.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:40.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:40.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:40.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:41.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:41.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:42.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:42.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:42.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:42.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:43.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:43.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:43.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:43.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:44.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:44.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:44.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:44.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:44.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:44.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:45.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:45.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:45.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:45.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:46.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:46.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:46.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:46.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:47.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:47.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:47.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:47.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:48.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:48.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:48.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:48.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:49.204 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:49.195+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:49.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:49.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:49.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:49.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:50.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:50.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:50.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:50.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:51.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:51.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:51.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:51.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:52.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:52.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:52.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:52.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:53.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:53.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:53.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:53.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:54.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:54.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:54.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:54.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:54.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:54.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:55.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:55.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:55.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:55.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:56.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:56.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:56.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:56.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:57.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:57.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:57.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:57.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:58.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:58.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:58.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:58.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:32:59.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:32:59.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:32:59.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:59.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:32:59.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:32:59.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:00.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:00.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:00.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:00.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:01.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:01.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:01.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:01.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:02.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:02.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:02.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:02.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:03.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:03.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:03.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:03.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:04.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:04.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:04.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:04.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:04.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:04.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:05.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:05.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:05.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:05.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:06.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:06.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:07.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:07.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:07.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:07.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:08.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:08.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:08.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:08.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:09.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:09.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:09.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:09.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:09.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:09.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:10.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:10.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:10.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:10.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:11.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:11.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:11.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:11.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:12.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:12.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:12.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:12.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:13.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:13.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:13.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:13.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:14.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:14.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:14.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:14.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:14.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:14.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:15.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:15.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:15.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:15.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:16.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:16.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:16.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:16.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:17.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:17.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:17.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:17.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:18.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:18.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:18.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:18.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:19.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:19.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:19.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:19.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:19.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:19.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:20.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:20.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:20.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:20.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:21.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:21.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:21.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:21.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:22.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:22.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:22.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:22.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:23.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:23.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:23.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:23.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:24.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:24.202+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:24.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:24.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:25.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:25.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:25.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:25.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:26.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:26.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:26.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:26.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:27.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:27.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:27.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:27.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:28.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:28.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:28.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:28.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:29.211 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:29.202+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:29.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:29.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:29.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:29.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:30.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:30.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:30.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:30.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:31.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:31.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:32.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:32.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:32.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:32.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:33.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:33.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:33.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:33.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:34.211 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:34.202+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:34.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:34.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:34.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:34.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:35.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:35.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:35.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:35.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:36.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:36.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:36.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:36.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:37.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:37.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:37.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:37.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:38.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:38.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:38.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:38.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:39.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:39.202+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:39.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:39.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:40.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:40.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:40.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:40.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:41.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:41.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:41.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:41.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:42.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:42.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:42.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:42.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:43.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:43.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:43.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:43.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:44.225 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:44.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:44.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:44.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:44.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:44.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:45.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:45.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:45.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:45.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:46.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:46.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:46.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:46.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:47.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:47.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:47.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:47.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:48.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:48.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:48.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:48.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:49.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:49.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:49.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:49.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:49.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:49.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:50.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:50.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:50.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:50.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:51.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:51.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:51.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:51.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:52.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:52.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:52.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:52.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:53.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:53.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:53.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:53.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:54.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:54.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:54.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:54.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:54.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:54.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:55.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:55.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:55.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:55.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:56.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:56.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:56.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:56.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:57.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:57.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:57.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:57.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:58.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:58.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:58.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:58.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:33:59.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:33:59.205+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:33:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:59.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:33:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:33:59.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:00.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:00.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:00.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:01.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:01.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:01.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:01.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:02.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:02.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:02.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:02.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:03.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:03.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:03.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:03.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:04.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:04.205+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:04.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:04.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:04.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:04.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:05.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:05.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:05.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:05.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:06.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:06.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:06.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:06.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:07.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:07.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:07.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:07.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:08.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:08.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:08.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:08.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:09.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:09.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:09.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:09.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:09.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:09.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:10.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:10.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:10.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:10.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:11.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:11.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:11.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:11.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:12.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:12.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:12.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:12.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:13.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:13.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:13.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:13.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:14.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:14.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:14.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:14.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:14.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:14.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:15.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:15.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:15.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:15.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:16.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:16.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:16.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:16.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:17.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:17.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:17.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:17.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:18.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:18.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:18.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:19.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:19.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:19.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:19.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:19.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:19.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:20.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:20.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:20.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:20.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:21.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:21.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:21.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:21.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:22.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:22.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:22.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:22.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:23.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:23.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:23.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:23.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:24.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:24.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:24.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:24.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:24.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:24.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:25.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:25.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:25.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:25.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:26.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:26.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:26.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:26.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:27.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:27.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:27.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:27.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:28.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:28.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:28.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:28.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:29.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:29.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:29.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:29.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:29.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:29.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:30.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:30.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:30.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:30.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:31.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:31.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:31.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:31.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:32.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:32.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:32.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:32.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:33.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:33.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:33.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:33.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:34.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:34.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:34.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:34.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:34.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:34.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:35.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:35.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:35.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:35.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:36.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:36.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:36.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:36.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:37.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:37.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:37.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:37.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:38.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:38.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:38.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:38.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:39.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:39.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:39.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:39.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:39.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:39.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:40.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:40.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:40.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:40.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:41.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:41.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:42.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:42.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:42.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:43.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:43.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:43.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:43.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:44.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:44.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:44.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:44.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:44.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:44.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:45.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:45.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:45.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:45.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:46.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:46.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:46.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:46.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:47.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:47.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:47.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:47.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:48.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:48.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:48.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:48.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:49.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:49.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:49.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:49.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:50.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:50.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:50.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:51.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:51.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:51.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:52.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:52.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:52.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:52.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:53.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:53.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:53.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:53.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:54.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:54.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:54.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:54.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:54.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:54.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:55.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:55.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:55.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:55.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:56.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:56.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:56.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:56.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:57.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:57.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:57.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:57.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:58.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:58.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:58.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:58.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:34:59.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:34:59.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:34:59.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:59.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:34:59.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:34:59.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:00.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:00.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:00.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:00.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:01.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:01.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:01.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:01.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:02.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:02.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:02.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:02.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:03.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:03.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:03.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:03.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:04.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:04.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:04.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:04.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:04.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:04.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:05.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:05.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:05.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:05.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:06.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:06.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:06.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:06.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:07.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:07.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:07.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:07.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:08.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:08.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:08.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:08.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:09.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:09.219+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:09.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:09.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:09.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:09.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:10.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:10.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:10.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:10.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:11.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:11.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:11.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:11.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:12.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:12.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:12.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:12.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:13.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:13.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:13.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:13.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:14.229 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:14.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:14.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:14.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:14.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:14.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:15.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:15.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:15.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:15.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:16.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:16.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:16.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:16.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:17.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:17.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:17.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:17.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:18.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:18.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:18.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:18.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:19.229 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:19.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:19.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:19.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:19.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:19.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:20.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:20.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:20.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:20.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:21.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:21.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:21.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:21.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:22.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:22.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:22.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:22.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:23.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:23.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:23.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:23.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:24.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:24.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:24.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:24.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:24.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:24.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:25.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:25.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:25.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:25.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:26.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:26.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:26.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:26.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:27.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:27.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:27.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:27.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:28.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:28.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:28.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:28.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:29.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:29.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:29.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:29.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:29.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:29.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:30.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:30.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:30.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:30.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:31.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:31.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:31.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:31.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:32.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:32.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:32.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:32.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:33.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:33.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:33.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:33.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:34.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:34.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:34.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:34.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:34.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:34.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:35.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:35.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:35.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:35.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:36.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:36.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:36.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:36.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:37.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:37.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:37.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:37.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:38.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:38.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:38.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:38.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:39.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:39.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:39.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:39.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:39.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:39.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:40.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:40.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:40.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:40.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:41.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:41.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:41.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:41.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:42.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:42.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:42.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:42.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:43.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:43.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:43.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:43.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:44.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:44.226+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:44.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:44.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:44.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:44.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:45.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:45.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:45.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:45.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:46.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:46.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:46.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:46.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:47.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:47.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:47.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:47.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:48.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:48.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:48.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:48.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:49.234 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:49.226+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:49.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:49.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:49.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:49.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:50.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:50.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:50.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:50.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:51.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:51.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:51.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:51.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:52.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:52.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:52.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:52.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:53.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:53.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:53.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:53.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:54.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:54.226+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:54.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:54.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:54.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:54.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:55.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:55.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:56.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:56.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:56.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:56.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:57.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:57.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:57.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:57.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:58.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:58.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:58.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:58.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:35:59.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:35:59.226+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:35:59.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:59.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:35:59.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:35:59.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:00.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:00.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:00.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:00.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:01.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:01.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:01.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:01.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:02.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:02.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:02.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:02.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:03.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:03.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:03.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:03.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:04.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:04.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:04.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:04.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:04.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:04.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:05.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:05.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:05.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:05.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:06.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:06.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:06.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:06.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:07.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:07.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:07.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:07.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:09.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:08.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:09.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:08.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:09.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:09.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:10.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:10.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:10.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:10.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:11.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:10.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:11.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:10.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:12.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:11.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:12.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:11.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:13.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:13.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:13.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:13.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:14.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:14.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:14.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:14.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:14.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:14.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:15.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:15.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:15.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:15.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:16.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:16.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:16.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:16.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:17.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:17.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:17.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:17.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:18.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:18.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:18.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:18.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:19.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:19.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:19.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:19.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:19.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:19.229+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:20.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:20.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:20.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:20.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:21.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:21.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:21.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:21.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:22.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:22.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:22.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:22.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:23.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:23.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:23.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:23.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:24.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:24.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:24.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:24.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:24.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:24.229+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:25.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:25.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:25.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:25.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:26.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:26.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:26.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:26.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:27.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:27.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:27.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:27.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:28.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:28.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:28.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:28.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:28.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:28.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:28.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:28.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:29.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:29.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:29.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:29.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:29.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:29.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:30.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:30.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:30.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:30.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:31.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:31.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:31.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:31.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:32.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:32.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:32.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:32.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:33.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:33.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:33.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:33.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:34.242 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:34.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:34.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:34.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:34.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:34.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:35.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:35.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:35.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:35.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:36.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:36.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:36.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:36.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:37.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:37.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:37.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:37.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:38.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:38.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:38.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:38.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:39.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:39.237+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:39.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:39.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:40.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:40.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:40.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:40.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:41.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:41.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:41.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:41.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:42.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:42.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:42.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:42.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:43.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:43.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:43.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:43.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:44.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:44.237+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:44.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:44.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:44.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:44.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:45.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:45.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:45.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:45.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:46.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:46.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:46.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:46.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:47.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:47.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:47.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:47.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:48.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:48.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:48.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:48.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:49.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:49.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:49.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:49.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:49.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:49.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:50.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:50.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:50.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:50.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:51.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:51.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:51.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:51.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:52.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:52.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:52.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:52.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:53.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:53.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:53.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:53.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:54.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:54.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:54.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:54.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:54.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:54.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:55.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:55.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:55.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:55.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:56.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:56.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:56.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:56.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:57.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:57.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:57.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:57.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:58.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:58.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:58.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:58.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:36:59.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:36:59.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:36:59.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:59.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:36:59.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:36:59.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:00.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:00.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:00.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:00.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:01.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:01.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:01.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:01.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:02.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:02.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:02.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:02.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:03.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:03.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:03.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:03.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:04.247 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:04.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:04.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:04.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:04.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:04.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:05.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:05.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:05.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:05.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:06.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:06.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:06.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:06.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:07.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:07.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:07.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:07.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:08.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:08.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:08.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:08.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:09.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:09.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:09.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:09.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:09.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:09.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:10.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:10.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:10.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:10.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:11.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:11.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:11.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:11.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:12.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:12.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:12.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:12.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:13.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:13.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:13.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:13.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:14.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:14.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:14.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:14.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:14.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:14.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:15.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:15.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:15.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:15.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:16.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:16.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:16.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:16.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:17.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:17.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:17.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:17.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:18.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:18.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:18.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:18.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:19.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:19.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:19.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:19.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:19.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:19.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:20.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:20.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:21.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:21.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:21.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:21.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:22.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:22.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:22.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:22.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:23.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:23.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:23.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:23.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:24.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:24.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:24.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:24.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:24.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:24.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:25.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:25.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:25.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:25.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:26.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:26.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:26.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:26.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:27.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:27.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:27.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:27.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:28.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:28.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:28.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:28.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:29.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:29.243+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:29.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:29.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:29.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:29.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:31.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:31.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:31.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:31.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:32.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:32.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:32.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:32.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:32.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:32.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:32.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:32.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:33.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:33.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:33.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:33.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:34.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:34.243+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:34.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:34.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:34.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:34.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:35.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:35.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:35.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:35.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:36.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:36.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:36.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:36.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:37.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:37.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:37.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:37.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:38.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:38.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:38.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:38.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:39.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:39.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:39.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:39.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:39.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:39.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:40.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:40.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:40.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:40.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:41.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:41.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:41.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:41.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:42.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:42.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:42.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:42.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:43.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:43.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:43.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:43.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:44.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:44.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:44.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:44.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:44.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:44.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:45.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:45.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:45.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:45.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:46.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:46.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:46.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:46.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:47.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:47.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:47.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:47.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:48.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:48.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:48.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:48.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:49.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:49.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:49.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:49.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:49.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:49.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:50.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:50.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:50.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:50.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:51.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:51.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:51.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:51.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:52.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:52.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:52.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:52.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:53.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:53.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:53.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:53.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:54.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:54.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:54.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:54.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:54.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:54.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:55.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:55.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:56.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:56.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:56.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:56.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:57.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:57.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:57.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:57.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:58.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:58.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:58.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:58.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:37:59.257 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:37:59.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:37:59.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:59.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:37:59.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:37:59.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:00.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:00.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:00.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:00.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:01.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:01.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:01.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:01.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:02.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:02.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:02.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:02.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:03.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:03.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:03.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:03.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:04.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:04.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:04.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:04.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:04.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:04.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:05.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:05.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:05.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:05.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:06.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:06.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:06.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:06.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:07.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:07.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:07.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:07.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:08.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:08.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:08.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:08.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:09.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:09.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:09.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:09.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:09.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:09.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:10.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:10.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:10.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:10.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:11.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:11.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:11.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:11.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:12.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:12.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:13.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:13.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:13.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:13.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:14.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:14.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:14.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:14.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:14.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:14.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:15.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:15.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:15.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:15.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:16.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:16.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:16.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:16.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:17.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:17.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:17.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:17.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:18.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:18.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:19.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:19.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:19.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:19.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:19.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:19.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:20.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:20.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:20.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:20.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:21.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:21.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:21.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:21.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:22.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:22.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:22.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:22.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:23.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:23.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:23.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:23.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:24.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:24.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:24.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:24.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:24.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:24.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:25.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:25.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:25.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:25.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:26.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:26.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:26.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:26.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:27.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:27.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:27.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:27.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:28.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:28.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:28.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:28.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:29.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:29.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:29.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:29.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:29.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:29.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:30.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:30.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:30.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:30.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:31.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:31.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:31.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:31.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:32.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:32.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:32.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:32.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:33.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:33.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:33.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:33.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:34.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:34.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:34.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:34.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:34.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:34.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:35.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:35.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:35.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:35.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:36.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:36.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:36.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:36.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:37.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:37.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:37.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:37.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:38.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:38.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:38.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:38.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:39.263 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:39.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:39.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:39.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:39.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:39.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:40.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:40.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:40.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:40.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:41.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:41.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:41.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:41.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:42.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:42.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:42.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:42.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:43.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:43.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:43.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:43.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:44.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:44.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:44.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:44.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:44.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:44.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:45.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:45.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:45.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:45.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:46.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:46.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:46.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:46.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:47.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:47.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:47.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:47.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:48.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:48.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:48.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:48.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:49.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:49.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:49.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:49.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:49.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:49.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:50.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:50.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:50.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:50.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:51.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:51.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:51.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:51.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:52.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:52.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:52.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:52.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:53.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:53.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:53.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:53.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:54.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:54.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:54.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:54.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:54.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:54.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:55.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:55.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:55.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:55.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:56.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:56.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:56.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:56.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:57.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:57.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:57.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:57.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:58.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:58.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:58.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:58.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:38:59.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:38:59.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:38:59.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:59.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:38:59.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:38:59.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:00.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:00.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:00.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:00.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:01.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:01.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:01.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:01.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:02.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:02.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:02.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:02.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:03.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:03.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:03.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:03.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:04.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:04.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:04.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:04.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:04.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:04.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:05.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:05.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:05.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:05.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:06.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:06.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:06.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:06.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:07.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:07.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:07.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:07.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:08.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:08.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:08.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:08.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:09.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:09.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:09.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:09.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:09.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:09.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:10.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:10.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:10.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:10.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:11.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:11.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:11.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:11.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:12.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:12.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:12.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:12.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:13.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:13.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:13.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:13.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:14.269 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:14.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:14.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:14.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:14.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:14.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:15.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:15.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:15.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:15.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:16.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:16.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:16.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:16.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:17.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:17.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:18.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:18.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:18.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:18.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:19.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:19.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:19.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:19.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:19.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:19.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:20.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:20.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:20.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:20.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:21.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:21.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:21.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:21.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:22.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:22.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:22.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:23.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:23.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:23.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:23.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:24.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:24.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:24.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:24.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:24.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:24.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:25.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:25.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:25.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:25.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:26.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:26.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:26.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:26.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:27.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:27.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:27.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:27.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:28.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:28.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:28.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:28.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:29.271 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:29.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:29.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:29.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:29.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:29.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:30.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:30.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:30.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:30.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:31.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:31.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:31.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:31.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:32.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:32.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:32.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:32.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:33.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:33.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:29:00.485331+0000 (oldest deadline 2019-10-17T23:29:20.485331+0000) 2019-10-17T23:39:33.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:33.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:34.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:34.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:34.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:34.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:35.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:35.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:36.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:36.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:37.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:37.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:38.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:38.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:39.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:39.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:39.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:39.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:40.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:40.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:41.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:41.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:42.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:42.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:43.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:43.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:44.274 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:44.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:44.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:44.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:45.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:45.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:46.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:47.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:47.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:48.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:48.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:49.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:49.267+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:49.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:49.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:50.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:50.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:51.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:51.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:52.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:52.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:53.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:53.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:54.276 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:54.267+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:54.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:54.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:55.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:55.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:56.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:56.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:57.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:57.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:39:57.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:57.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:58.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:58.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:39:58.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:58.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:39:59.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:39:59.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:39:59.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:59.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:39:59.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:39:59.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:00.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:00.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:00.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:00.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:01.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:01.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:01.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:01.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:02.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:02.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:02.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:02.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:03.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:03.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:03.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:03.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:04.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:04.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:04.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:04.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:04.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:04.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:05.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:05.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:05.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:05.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:06.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:06.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:06.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:06.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:07.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:07.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:07.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:07.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:08.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:08.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:08.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:08.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:09.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:09.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:09.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:09.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:09.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:09.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:10.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:10.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:10.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:10.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:11.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:11.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:11.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:11.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:12.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:12.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:12.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:12.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:13.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:13.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:13.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:13.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:14.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:14.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:14.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:14.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:14.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:14.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:15.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:15.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:15.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:15.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:16.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:16.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:16.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:16.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:17.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:17.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:17.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:17.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:18.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:18.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:18.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:18.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:19.284 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:19.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:19.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:19.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:19.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:19.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:20.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:20.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:20.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:20.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:21.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:21.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:21.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:21.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:22.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:22.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:23.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:23.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:23.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:23.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:24.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:24.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:24.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:24.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:24.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:24.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:25.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:25.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:25.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:25.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:26.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:26.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:26.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:26.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:27.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:27.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:27.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:27.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:28.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:28.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:28.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:28.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:29.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:29.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:29.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:29.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:29.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:29.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:30.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:30.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:30.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:30.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:31.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:31.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:31.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:31.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:32.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:32.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:32.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:32.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:33.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:33.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:33.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:33.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:34.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:34.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:34.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:34.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:34.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:34.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:35.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:35.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:35.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:35.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:36.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:36.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:36.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:36.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:37.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:37.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:37.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:37.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:38.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:38.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:38.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:38.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:39.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:39.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:39.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:39.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:39.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:39.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:40.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:40.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:40.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:40.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:41.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:41.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:41.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:41.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:42.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:42.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:42.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:42.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:43.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:43.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:44.284 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:44.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:44.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:44.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:44.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:44.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:45.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:45.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:45.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:45.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:46.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:46.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:46.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:46.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:47.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:47.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:47.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:47.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:48.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:48.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:48.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:48.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:49.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:49.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:49.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:49.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:49.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:49.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:50.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:50.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:50.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:50.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:51.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:51.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:51.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:51.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:52.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:52.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:52.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:52.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:53.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:53.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:53.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:53.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:54.286 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:54.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:54.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:54.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:54.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:54.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:55.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:55.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:55.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:55.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:56.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:56.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:56.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:56.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:57.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:57.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:57.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:57.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:58.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:58.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:58.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:58.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:40:59.287 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:40:59.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:40:59.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:59.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:40:59.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:40:59.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:00.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:00.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:01.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:01.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:01.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:01.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:02.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:02.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:02.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:02.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:03.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:03.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:03.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:03.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:04.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:04.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:04.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:04.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:04.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:04.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:05.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:05.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:05.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:05.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:06.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:06.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:07.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:07.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:07.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:07.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:08.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:08.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:08.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:08.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:09.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:09.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:09.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:09.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:09.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:09.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:10.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:10.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:10.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:10.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:11.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:11.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:11.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:11.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:12.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:12.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:12.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:12.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:13.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:13.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:13.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:13.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:14.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:14.281+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:14.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:14.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:14.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:14.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:15.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:15.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:15.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:15.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:16.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:16.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:16.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:16.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:17.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:17.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:17.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:17.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:18.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:18.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:18.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:18.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:19.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:19.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:19.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:19.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:19.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:19.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:20.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:20.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:20.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:20.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:21.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:21.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:21.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:21.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:22.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:22.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:22.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:22.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:23.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:23.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:23.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:23.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:24.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:24.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:24.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:24.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:24.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:24.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:25.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:25.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:25.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:25.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:26.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:26.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:26.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:26.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:27.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:27.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:27.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:27.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:28.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:28.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:28.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:28.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:29.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:29.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:29.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:29.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:29.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:29.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:30.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:30.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:30.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:30.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:31.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:31.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:31.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:31.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:32.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:32.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:32.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:32.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:33.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:33.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:33.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:33.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:34.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:34.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:34.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:34.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:34.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:34.288+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:35.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:35.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:35.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:35.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:36.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:36.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:36.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:36.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:37.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:37.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:37.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:37.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:38.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:38.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:38.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:38.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:39.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:39.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:39.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:39.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:39.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:39.288+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:40.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:40.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:40.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:40.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:41.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:41.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:42.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:42.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:42.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:42.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:43.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:43.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:43.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:43.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:44.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:44.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:44.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:44.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:44.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:44.288+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:45.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:45.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:45.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:45.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:46.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:46.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:46.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:46.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:47.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:47.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:47.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:47.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:48.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:48.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:48.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:48.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:49.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:49.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:49.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:49.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:49.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:49.288+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:50.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:50.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:50.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:50.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:51.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:51.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:51.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:51.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:52.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:52.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:52.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:52.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:53.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:53.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:53.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:53.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:54.297 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:54.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:41:54.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:54.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:54.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:54.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:55.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:55.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:55.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:55.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:56.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:56.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:56.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:56.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:57.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:57.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:57.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:57.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:58.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:58.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:58.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:58.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:59.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:59.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:41:59.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:41:59.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:41:59.297 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:41:59.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:00.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:00.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:00.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:00.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:01.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:01.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:01.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:01.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:02.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:02.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:02.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:02.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:03.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:03.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:03.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:03.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:04.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:04.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:04.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:04.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:04.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:04.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:05.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:05.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:05.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:05.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:06.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:06.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:06.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:06.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:07.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:07.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:07.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:07.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:08.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:08.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:08.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:08.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:09.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:09.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:09.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:09.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:09.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:09.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:10.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:10.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:10.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:10.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:11.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:11.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:11.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:11.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:12.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:12.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:13.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:13.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:13.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:13.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:14.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:14.291+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:14.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:14.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:14.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:14.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:15.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:15.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:15.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:15.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:16.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:16.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:16.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:16.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:17.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:17.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:17.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:17.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:18.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:18.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:18.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:18.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:19.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:19.291+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:19.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:19.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:19.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:19.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:20.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:20.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:20.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:20.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:21.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:21.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:21.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:21.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:22.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:22.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:22.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:22.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:23.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:23.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:23.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:23.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:24.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:24.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:24.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:24.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:25.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:25.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:25.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:26.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:26.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:26.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:27.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:27.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:27.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:27.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:28.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:28.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:28.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:28.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:29.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:29.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:29.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:29.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:29.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:29.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:30.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:30.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:30.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:30.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:31.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:31.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:31.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:31.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:32.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:32.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:32.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:32.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:33.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:33.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:33.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:33.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:34.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:34.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:34.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:34.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:34.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:34.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:35.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:35.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:35.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:35.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:36.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:36.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:36.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:36.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:37.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:37.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:37.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:37.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:38.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:38.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:39.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:39.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:39.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:39.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:39.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:39.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:40.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:40.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:40.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:40.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:41.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:41.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:41.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:41.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:42.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:42.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:42.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:43.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:43.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:43.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:43.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:44.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:44.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:44.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:44.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:44.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:44.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:45.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:45.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:45.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:45.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:46.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:46.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:46.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:46.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:47.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:47.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:47.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:47.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:48.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:48.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:48.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:48.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:49.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:49.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:49.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:49.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:49.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:49.298+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:50.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:50.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:50.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:50.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:51.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:51.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:51.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:51.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:52.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:52.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:52.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:52.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:53.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:53.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:53.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:53.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:54.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:54.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:54.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:54.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:54.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:54.298+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:42:55.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:55.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:55.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:55.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:56.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:56.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:56.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:56.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:56.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:56.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:56.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:56.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:58.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:58.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:58.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:58.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:59.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:59.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:42:59.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:42:59.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:42:59.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:42:59.298+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:00.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:00.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:00.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:00.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:01.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:01.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:01.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:01.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:02.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:02.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:02.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:02.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:03.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:03.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:03.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:03.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:04.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:04.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:04.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:04.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:04.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:04.298+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:05.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:04.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:05.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:04.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:05.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:05.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:05.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:06.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:06.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:06.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:06.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:07.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:07.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:07.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:07.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:08.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:08.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:08.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:08.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:09.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:09.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:10.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:10.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:10.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:10.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:11.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:11.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:11.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:11.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:12.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:12.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:12.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:12.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:13.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:13.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:13.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:13.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:14.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:14.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:14.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:14.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:14.310 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:14.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:15.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:15.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:15.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:15.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:16.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:16.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:16.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:16.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:17.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:17.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:17.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:17.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:18.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:18.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:18.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:18.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:19.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:19.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:19.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:19.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:19.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:19.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:20.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:20.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:20.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:20.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:21.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:21.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:22.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:22.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:22.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:22.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:23.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:23.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:24.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:24.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:24.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:24.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:24.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:24.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:25.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:25.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:25.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:25.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:26.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:26.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:26.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:26.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:27.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:27.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:27.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:27.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:28.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:28.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:28.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:28.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:29.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:29.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:29.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:29.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:30.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:30.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:30.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:30.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:31.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:31.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:31.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:31.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:32.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:32.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:32.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:32.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:33.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:33.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:33.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:33.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:34.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:34.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:34.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:34.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:34.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:34.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:35.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:35.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:35.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:35.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:36.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:36.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:36.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:36.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:37.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:37.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:37.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:37.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:38.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:38.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:38.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:38.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:39.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:39.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:39.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:39.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:39.314 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:39.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:40.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:40.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:40.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:40.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:41.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:41.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:41.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:41.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:42.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:42.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:42.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:42.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:43.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:43.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:43.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:43.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:44.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:44.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:44.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:44.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:44.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:44.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:45.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:45.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:45.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:45.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:46.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:46.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:46.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:46.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:47.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:47.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:47.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:47.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:48.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:48.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:48.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:48.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:49.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:49.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:49.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:49.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:49.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:49.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:50.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:50.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:50.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:50.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:51.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:51.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:51.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:51.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:52.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:52.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:52.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:52.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:53.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:53.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:53.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:53.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:54.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:54.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:43:54.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:54.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:54.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:54.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:55.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:55.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:55.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:55.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:56.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:56.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:56.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:56.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:57.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:57.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:57.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:57.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:58.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:58.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:58.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:58.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:59.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:43:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:43:59.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:43:59.318 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:43:59.312+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:00.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:00.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:00.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:00.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:01.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:01.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:01.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:01.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:02.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:02.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:02.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:02.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:03.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:03.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:03.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:03.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:04.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:04.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:04.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:04.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:04.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:04.312+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:05.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:05.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:05.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:05.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:06.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:06.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:06.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:06.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:07.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:07.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:07.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:07.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:08.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:08.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:08.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:08.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:09.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:09.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:09.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:09.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:09.312+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:10.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:10.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:10.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:10.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:11.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:11.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:11.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:11.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:12.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:12.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:12.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:12.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:13.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:13.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:13.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:13.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:14.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:14.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:14.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:14.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:14.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:14.312+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:15.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:15.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:15.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:15.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:16.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:16.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:16.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:16.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:17.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:17.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:17.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:17.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:18.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:18.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:18.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:18.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:19.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:19.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:19.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:19.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:19.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:19.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:20.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:20.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:20.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:20.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:21.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:21.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:21.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:21.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:22.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:22.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:22.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:22.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:23.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:23.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:23.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:23.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:24.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:24.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:24.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:24.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:24.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:24.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:25.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:25.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:25.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:25.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:26.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:26.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:26.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:26.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:27.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:27.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:27.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:28.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:28.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:28.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:28.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:29.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:29.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:29.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:29.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:29.323 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:29.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:30.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:30.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:30.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:30.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:31.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:31.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:31.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:31.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:32.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:32.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:32.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:32.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:33.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:33.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:33.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:33.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:34.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:34.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:34.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:34.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:34.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:34.315+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:35.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:35.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:35.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:35.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:35.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:35.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:36.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:36.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:36.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:36.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:38.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:38.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:38.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:38.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:38.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:38.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:38.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:38.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:39.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:39.315+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:40.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:40.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:40.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:40.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:41.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:41.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:41.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:41.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:42.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:42.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:42.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:42.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:43.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:43.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:43.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:43.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:44.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:44.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:44.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:44.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:44.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:44.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:45.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:45.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:45.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:45.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:46.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:46.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:46.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:46.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:47.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:47.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:47.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:47.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:48.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:48.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:48.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:48.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:49.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:49.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:49.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:49.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:49.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:49.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:50.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:50.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:50.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:50.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:51.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:51.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:51.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:51.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:52.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:52.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:52.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:52.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:53.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:53.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:53.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:53.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:54.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:54.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:54.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:54.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:54.327 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:54.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:44:55.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:55.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:55.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:55.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:56.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:56.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:56.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:56.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:57.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:57.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:57.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:57.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:58.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:58.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:58.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:58.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:59.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:59.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:44:59.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:44:59.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:44:59.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:44:59.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:00.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:00.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:00.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:00.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:01.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:01.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:01.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:01.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:02.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:02.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:02.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:02.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:03.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:03.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:03.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:03.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:04.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:04.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:04.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:04.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:04.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:04.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:05.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:05.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:05.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:05.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:06.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:06.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:06.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:06.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:07.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:07.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:07.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:07.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:08.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:08.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:08.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:08.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:09.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:09.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:09.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:09.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:09.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:09.322+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:10.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:10.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:10.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:10.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:11.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:11.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:11.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:11.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:12.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:12.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:12.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:12.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:13.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:13.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:13.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:13.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:14.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:14.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:14.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:14.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:14.331 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:14.322+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:15.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:15.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:15.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:15.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:16.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:16.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:16.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:16.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:17.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:17.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:17.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:17.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:18.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:18.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:18.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:18.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:19.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:19.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:19.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:19.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:19.331 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:19.322+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:20.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:20.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:20.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:20.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:21.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:21.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:21.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:21.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:22.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:22.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:22.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:22.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:23.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:23.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:23.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:23.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:24.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:24.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:24.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:24.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:24.322+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:25.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:25.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:25.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:25.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:26.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:26.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:26.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:26.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:27.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:27.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:27.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:27.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:28.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:28.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:28.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:28.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:29.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:29.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:29.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:29.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:29.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:29.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:30.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:30.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:30.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:30.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:31.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:31.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:31.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:31.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:32.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:32.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:32.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:32.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:33.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:33.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:33.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:33.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:34.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:34.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:34.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:34.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:34.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:34.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:35.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:35.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:35.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:35.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:36.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:36.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:36.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:36.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:37.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:37.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:37.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:37.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:38.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:38.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:38.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:38.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:39.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:39.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:39.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:39.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:39.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:39.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:40.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:40.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:40.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:40.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:41.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:41.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:41.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:41.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:42.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:42.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:42.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:42.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:43.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:43.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:43.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:43.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:44.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:44.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:44.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:44.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:44.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:44.325+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:45.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:45.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:45.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:45.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:46.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:46.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:46.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:46.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:47.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:47.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:47.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:47.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:48.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:48.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:48.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:48.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:49.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:49.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:49.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:49.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:49.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:49.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:50.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:50.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:50.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:50.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:50.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:45:50.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:45:50.110969+0000) 2019-10-17T23:45:51.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:51.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:51.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:51.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:51.611 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:45:51.604+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:45:51.608411+0000) 2019-10-17T23:45:52.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:52.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:52.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:52.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:52.551 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:45:52.541+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:45:52.548069+0000) 2019-10-17T23:45:53.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:53.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:53.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:53.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:54.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:54.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:54.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:54.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:54.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:54.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:45:54.591 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:45:54.581+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:45:54.588477+0000) 2019-10-17T23:45:55.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:55.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:55.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:55.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:56.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:56.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:56.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:56.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:57.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:57.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:57.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:57.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:58.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:58.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:58.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:58.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:59.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:45:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:45:59.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:45:59.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:45:59.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:00.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:46:00.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:00.111094+0000) 2019-10-17T23:46:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:00.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:00.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:01.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:01.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:01.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:01.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:01.612 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:46:01.604+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:01.608587+0000) 2019-10-17T23:46:02.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:02.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:02.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:02.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:02.551 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:46:02.545+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:02.548169+0000) 2019-10-17T23:46:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:03.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:03.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:04.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:04.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:04.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:04.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:04.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:04.333+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:04.592 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:46:04.585+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:04.588596+0000) 2019-10-17T23:46:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:05.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:05.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:05.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:06.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:06.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:06.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:06.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:07.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:07.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:07.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:07.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:08.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:08.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:08.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:08.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:09.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:09.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:09.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:09.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:09.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:09.333+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:10.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:46:10.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:10.111212+0000) 2019-10-17T23:46:10.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:10.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:10.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:10.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:11.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:11.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:11.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:11.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:11.612 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:46:11.604+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:11.608757+0000) 2019-10-17T23:46:12.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:12.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:12.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:12.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:12.551 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:46:12.545+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:12.548266+0000) 2019-10-17T23:46:13.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:13.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:13.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:13.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:14.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:14.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:14.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:14.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:14.342 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:14.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:14.592 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:46:14.585+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:14.588711+0000) 2019-10-17T23:46:15.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:15.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:15.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:15.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:16.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:16.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:16.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:16.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:17.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:17.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:17.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:17.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:18.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:18.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:18.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:18.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:19.342 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:19.336+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:19.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:19.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:19.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:19.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:20.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:46:20.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:20.111325+0000) 2019-10-17T23:46:20.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:20.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:20.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:20.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:21.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:21.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:21.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:21.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:21.612 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:46:21.604+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:21.608906+0000) 2019-10-17T23:46:22.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:22.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:22.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:22.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:22.551 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:46:22.544+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:22.548368+0000) 2019-10-17T23:46:23.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:23.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:23.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:23.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:24.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:24.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:24.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:24.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:24.343 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:24.336+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:24.592 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:46:24.584+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:24.588824+0000) 2019-10-17T23:46:25.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:25.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:25.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:25.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:26.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:26.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:26.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:26.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:27.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:27.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:27.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:27.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:28.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:28.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:28.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:28.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:29.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:29.336+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:29.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:29.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:29.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:29.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:30.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:46:30.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:30.111451+0000) 2019-10-17T23:46:30.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:30.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:30.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:30.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:31.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:31.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:31.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:31.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:31.612 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:46:31.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:31.609069+0000) 2019-10-17T23:46:32.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:32.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:32.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:32.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:32.551 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:46:32.544+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:32.548467+0000) 2019-10-17T23:46:33.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:33.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:33.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:33.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:34.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:34.336+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:34.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:34.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:34.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:34.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:34.592 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:46:34.584+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:34.588944+0000) 2019-10-17T23:46:35.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:35.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:35.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:35.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:36.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:36.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:36.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:36.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:37.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:37.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:37.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:37.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:38.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:38.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:38.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:38.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:39.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:39.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:39.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:39.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:39.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:39.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:40.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:46:40.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:40.111566+0000) 2019-10-17T23:46:40.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:40.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:40.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:40.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:41.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:41.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:41.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:41.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:41.612 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:46:41.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:41.609250+0000) 2019-10-17T23:46:42.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:42.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:42.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:42.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:42.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:46:42.544+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:42.548567+0000) 2019-10-17T23:46:43.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:43.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:43.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:43.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:44.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:44.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:44.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:44.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:44.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:44.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:44.592 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:46:44.584+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:44.589260+0000) 2019-10-17T23:46:45.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:45.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:45.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:45.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:46.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:46.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:46.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:46.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:47.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:47.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:47.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:47.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:48.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:48.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:48.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:48.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:49.347 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:49.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:49.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:49.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:49.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:49.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:50.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:46:50.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:50.111681+0000) 2019-10-17T23:46:50.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:50.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:50.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:50.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:51.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:51.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:51.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:51.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:51.612 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:46:51.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:51.609434+0000) 2019-10-17T23:46:52.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:52.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:52.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:52.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:52.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:46:52.543+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:52.548670+0000) 2019-10-17T23:46:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:53.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:53.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:54.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:54.339+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:54.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:54.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:54.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:54.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:54.592 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:46:54.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:46:54.589478+0000) 2019-10-17T23:46:55.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:55.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:55.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:55.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:56.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:56.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:56.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:56.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:57.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:57.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:57.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:57.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:58.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:58.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:58.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:58.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:46:59.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:46:59.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:46:59.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:59.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:46:59.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:46:59.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:00.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:47:00.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:00.111800+0000) 2019-10-17T23:47:00.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:00.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:00.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:00.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:01.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:01.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:01.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:01.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:01.613 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:47:01.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:01.609617+0000) 2019-10-17T23:47:02.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:02.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:02.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:02.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:02.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:47:02.543+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:02.548770+0000) 2019-10-17T23:47:03.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:03.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:03.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:03.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:04.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:04.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:04.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:04.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:04.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:04.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:04.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:47:04.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:04.589681+0000) 2019-10-17T23:47:05.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:05.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:05.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:05.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:06.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:06.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:06.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:06.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:07.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:07.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:07.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:07.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:08.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:08.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:08.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:08.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:09.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:09.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:09.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:09.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:09.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:09.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:10.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:47:10.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:10.111921+0000) 2019-10-17T23:47:10.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:10.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:10.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:10.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:11.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:11.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:11.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:11.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:11.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:47:11.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:11.609783+0000) 2019-10-17T23:47:12.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:47:12.543+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:12.548874+0000) 2019-10-17T23:47:12.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:12.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:12.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:12.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:13.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:13.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:13.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:13.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:14.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:14.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:14.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:14.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:14.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:14.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:14.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:47:14.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:14.589794+0000) 2019-10-17T23:47:15.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:15.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:15.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:15.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:16.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:16.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:16.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:16.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:17.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:17.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:17.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:17.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:18.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:18.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:18.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:18.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:19.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:19.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:19.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:19.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:19.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:19.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:20.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:47:20.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:20.112039+0000) 2019-10-17T23:47:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:20.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:20.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:20.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:21.613 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:47:21.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:21.609986+0000) 2019-10-17T23:47:21.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:21.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:21.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:21.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:22.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:47:22.543+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:22.548972+0000) 2019-10-17T23:47:22.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:22.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:22.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:22.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:23.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:23.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:23.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:23.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:24.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:24.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:24.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:47:24.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:24.589899+0000) 2019-10-17T23:47:24.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:24.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:24.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:24.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:25.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:25.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:25.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:25.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:26.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:26.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:26.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:26.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:27.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:27.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:27.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:27.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:28.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:28.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:28.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:28.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:29.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:29.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:29.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:29.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:29.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:29.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:30.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:47:30.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:30.112152+0000) 2019-10-17T23:47:30.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:30.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:30.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:30.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:31.613 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:47:31.607+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:31.610168+0000) 2019-10-17T23:47:31.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:31.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:31.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:31.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:32.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:47:32.542+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:32.549069+0000) 2019-10-17T23:47:32.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:32.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:32.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:32.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:33.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:33.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:33.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:33.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:34.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:34.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:34.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:47:34.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:34.590009+0000) 2019-10-17T23:47:34.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:34.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:34.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:34.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:35.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:35.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:35.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:35.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:36.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:36.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:36.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:36.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:37.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:37.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:37.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:37.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:38.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:38.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:38.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:38.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:39.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:39.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:39.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:39.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:39.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:39.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:40.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:47:40.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:40.112271+0000) 2019-10-17T23:47:40.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:40.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:40.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:40.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:41.613 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:47:41.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:41.610377+0000) 2019-10-17T23:47:41.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:41.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:41.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:41.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:42.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:47:42.542+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:42.549171+0000) 2019-10-17T23:47:42.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:42.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:42.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:42.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:43.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:43.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:43.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:43.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:44.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:44.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:44.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:47:44.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:44.590111+0000) 2019-10-17T23:47:44.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:44.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:44.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:44.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:45.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:45.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:45.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:45.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:46.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:46.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:46.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:46.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:47.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:47.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:47.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:47.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:48.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:48.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:48.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:48.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:49.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:49.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:49.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:49.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:49.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:49.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:50.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:47:50.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:50.112388+0000) 2019-10-17T23:47:50.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:50.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:50.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:50.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:51.614 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:47:51.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:51.610576+0000) 2019-10-17T23:47:51.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:51.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:51.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:51.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:52.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:47:52.542+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:52.549275+0000) 2019-10-17T23:47:52.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:52.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:52.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:52.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:53.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:53.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:53.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:53.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:54.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:54.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:54.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:47:54.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:47:54.590218+0000) 2019-10-17T23:47:54.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:54.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:54.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:54.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:55.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:55.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:55.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:55.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:56.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:56.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:56.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:56.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:57.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:57.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:57.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:57.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:58.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:58.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:58.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:58.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:47:59.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:47:59.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:47:59.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:59.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:47:59.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:47:59.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:00.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:48:00.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:00.112506+0000) 2019-10-17T23:48:00.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:00.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:00.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:00.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:01.614 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:48:01.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:01.610720+0000) 2019-10-17T23:48:01.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:01.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:01.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:01.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:02.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:48:02.541+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:02.549376+0000) 2019-10-17T23:48:02.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:02.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:02.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:02.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:03.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:03.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:03.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:03.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:04.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:04.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:04.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:48:04.585+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:04.590333+0000) 2019-10-17T23:48:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:04.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:04.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:05.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:05.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:05.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:05.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:06.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:06.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:06.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:06.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:07.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:07.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:07.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:07.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:08.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:08.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:08.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:08.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:09.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:09.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:09.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:09.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:09.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:09.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:10.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:48:10.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:10.112625+0000) 2019-10-17T23:48:10.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:10.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:10.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:10.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:11.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:11.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:11.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:11.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:11.614 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:48:11.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:11.610929+0000) 2019-10-17T23:48:12.553 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:48:12.545+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:12.549474+0000) 2019-10-17T23:48:12.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:12.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:12.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:12.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:13.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:13.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:13.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:13.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:14.369 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:14.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:14.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:14.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:14.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:14.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:14.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:48:14.585+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:14.590463+0000) 2019-10-17T23:48:15.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:15.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:15.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:15.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:16.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:16.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:16.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:16.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:17.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:17.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:17.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:17.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:18.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:18.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:18.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:18.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:19.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:19.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:19.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:19.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:19.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:19.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:20.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:48:20.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:20.112741+0000) 2019-10-17T23:48:20.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:20.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:20.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:20.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:21.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:21.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:21.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:21.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:21.614 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:48:21.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:21.611107+0000) 2019-10-17T23:48:22.552 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:48:22.549+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:22.549580+0000) 2019-10-17T23:48:22.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:22.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:22.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:22.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:23.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:23.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:23.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:23.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:24.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:24.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:24.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:24.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:24.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:24.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:24.593 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:48:24.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:24.590566+0000) 2019-10-17T23:48:25.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:25.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:25.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:25.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:26.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:26.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:26.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:26.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:27.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:27.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:27.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:27.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:28.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:28.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:28.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:28.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:29.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:29.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:29.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:29.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:29.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:29.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:30.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:48:30.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:30.112856+0000) 2019-10-17T23:48:30.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:30.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:30.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:30.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:31.614 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:48:31.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:31.611299+0000) 2019-10-17T23:48:31.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:31.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:31.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:31.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:32.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:48:32.549+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:32.549674+0000) 2019-10-17T23:48:32.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:32.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:32.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:32.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:33.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:33.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:33.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:33.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:34.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:34.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:34.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:48:34.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:34.590677+0000) 2019-10-17T23:48:34.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:34.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:34.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:34.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:35.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:35.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:35.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:35.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:36.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:36.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:37.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:37.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:37.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:37.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:38.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:38.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:38.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:38.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:39.374 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:39.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:39.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:39.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:40.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:48:40.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:40.112970+0000) 2019-10-17T23:48:40.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:40.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:40.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:40.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:41.614 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:48:41.610+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:41.611449+0000) 2019-10-17T23:48:41.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:41.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:41.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:41.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:42.553 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:48:42.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:42.549773+0000) 2019-10-17T23:48:42.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:42.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:42.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:42.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:43.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:43.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:43.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:43.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:44.374 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:44.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:44.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:48:44.588+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:44.590787+0000) 2019-10-17T23:48:44.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:44.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:44.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:44.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:45.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:45.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:45.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:45.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:46.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:46.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:46.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:46.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:47.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:47.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:47.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:47.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:48.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:48.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:48.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:48.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:49.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:49.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:49.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:49.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:49.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:49.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:50.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:48:50.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:50.113087+0000) 2019-10-17T23:48:50.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:50.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:50.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:50.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:51.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:48:51.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:51.611623+0000) 2019-10-17T23:48:51.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:51.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:51.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:51.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:52.553 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:48:52.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:52.549875+0000) 2019-10-17T23:48:52.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:52.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:52.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:52.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:53.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:53.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:53.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:53.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:54.376 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:54.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:54.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:48:54.588+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:48:54.590903+0000) 2019-10-17T23:48:54.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:54.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:54.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:54.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:55.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:55.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:55.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:55.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:56.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:56.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:56.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:56.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:57.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:57.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:57.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:57.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:58.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:58.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:58.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:58.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:48:59.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:48:59.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:48:59.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:59.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:48:59.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:48:59.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:00.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:49:00.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:00.113205+0000) 2019-10-17T23:49:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:00.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:00.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:01.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:49:01.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:01.611783+0000) 2019-10-17T23:49:01.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:01.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:01.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:01.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:02.553 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:49:02.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:02.549974+0000) 2019-10-17T23:49:02.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:02.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:02.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:02.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:03.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:03.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:03.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:03.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:04.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:04.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:04.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:49:04.588+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:04.591012+0000) 2019-10-17T23:49:04.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:04.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:04.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:04.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:05.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:05.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:05.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:05.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:06.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:06.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:06.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:06.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:07.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:07.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:07.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:07.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:08.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:08.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:08.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:08.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:09.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:09.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:09.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:09.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:09.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:09.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:10.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:49:10.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:10.113320+0000) 2019-10-17T23:49:10.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:10.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:10.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:10.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:11.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:49:11.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:11.611958+0000) 2019-10-17T23:49:11.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:11.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:11.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:11.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:12.553 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:49:12.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:12.550074+0000) 2019-10-17T23:49:13.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:13.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:13.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:13.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:13.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:13.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:13.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:13.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:14.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:14.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:14.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:49:14.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:14.591138+0000) 2019-10-17T23:49:14.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:14.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:14.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:14.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:15.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:15.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:15.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:15.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:16.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:16.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:16.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:16.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:17.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:17.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:17.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:17.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:18.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:18.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:18.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:18.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:19.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:19.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:19.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:19.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:19.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:19.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:20.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:49:20.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:20.113433+0000) 2019-10-17T23:49:21.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:21.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:21.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:21.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:21.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:49:21.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:21.612148+0000) 2019-10-17T23:49:22.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:22.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:22.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:22.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:22.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:49:22.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:22.550171+0000) 2019-10-17T23:49:23.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:23.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:23.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:23.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:24.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:24.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:24.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:24.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:24.384 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:24.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:24.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:49:24.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:24.591263+0000) 2019-10-17T23:49:25.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:25.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:25.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:25.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:26.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:26.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:26.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:26.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:27.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:27.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:27.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:27.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:28.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:28.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:28.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:28.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:29.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:29.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:29.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:29.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:29.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:29.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:30.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:49:30.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:30.113547+0000) 2019-10-17T23:49:30.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:30.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:30.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:30.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:31.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:31.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:31.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:31.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:31.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:49:31.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:31.612329+0000) 2019-10-17T23:49:32.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:32.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:32.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:32.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:32.553 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:49:32.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:32.550266+0000) 2019-10-17T23:49:33.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:33.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:33.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:33.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:34.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:34.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:34.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:34.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:34.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:34.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:34.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:49:34.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:34.591376+0000) 2019-10-17T23:49:35.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:35.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:35.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:35.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:36.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:36.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:36.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:36.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:37.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:37.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:37.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:37.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:38.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:38.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:38.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:38.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:39.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:39.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:39.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:39.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:39.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:39.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:40.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:49:40.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:40.113662+0000) 2019-10-17T23:49:40.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:40.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:40.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:40.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:41.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:41.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:41.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:41.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:41.615 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:49:41.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:41.612479+0000) 2019-10-17T23:49:42.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:42.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:42.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:42.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:42.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:49:42.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:42.550366+0000) 2019-10-17T23:49:43.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:43.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:43.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:43.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:44.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:44.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:44.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:44.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:44.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:44.383+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:44.594 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:49:44.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:44.591478+0000) 2019-10-17T23:49:45.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:45.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:45.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:45.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:46.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:46.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:46.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:46.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:47.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:47.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:47.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:47.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:48.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:48.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:48.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:48.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:49.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:49.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:49.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:49.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:49.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:49.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:50.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:49:50.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:50.113772+0000) 2019-10-17T23:49:50.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:50.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:50.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:50.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:51.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:51.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:51.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:51.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:51.616 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:49:51.609+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:51.612690+0000) 2019-10-17T23:49:52.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:52.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:52.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:52.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:52.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:49:52.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:52.550465+0000) 2019-10-17T23:49:53.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:53.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:53.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:53.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:54.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:54.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:54.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:54.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:54.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:54.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:49:54.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:49:54.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:49:54.591588+0000) 2019-10-17T23:49:55.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:55.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:55.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:55.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:56.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:56.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:56.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:56.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:57.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:57.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:57.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:57.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:58.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:58.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:58.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:58.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:59.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:59.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:49:59.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:49:59.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:49:59.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:49:59.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:00.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:50:00.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:00.113893+0000) 2019-10-17T23:50:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:00.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:00.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:01.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:01.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:01.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:01.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:01.616 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:50:01.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:01.612876+0000) 2019-10-17T23:50:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:02.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:02.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:02.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:50:02.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:02.550561+0000) 2019-10-17T23:50:03.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:03.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:03.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:03.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:04.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:04.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:04.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:04.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:04.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:04.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:04.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:50:04.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:04.591694+0000) 2019-10-17T23:50:05.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:05.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:05.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:06.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:06.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:06.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:06.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:07.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:07.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:07.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:07.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:08.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:08.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:39:36.910409+0000 (oldest deadline 2019-10-17T23:39:56.910409+0000) 2019-10-17T23:50:08.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:08.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:09.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:09.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:09.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:09.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:10.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:10.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:10.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:50:10.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:10.114006+0000) 2019-10-17T23:50:11.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:11.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:11.616 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:50:11.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:11.613060+0000) 2019-10-17T23:50:12.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:12.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:12.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:50:12.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:12.550657+0000) 2019-10-17T23:50:13.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:13.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:14.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:14.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:14.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:14.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:14.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:50:14.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:14.591821+0000) 2019-10-17T23:50:15.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:15.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:16.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:16.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:17.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:17.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:18.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:18.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:19.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:19.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:19.397 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:19.390+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:20.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:20.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:20.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:50:20.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:20.114118+0000) 2019-10-17T23:50:21.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:21.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:21.616 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:50:21.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:21.613265+0000) 2019-10-17T23:50:22.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:22.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:22.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:50:22.545+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:22.550751+0000) 2019-10-17T23:50:23.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:23.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:24.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:24.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:24.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:24.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:24.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:50:24.585+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:24.591937+0000) 2019-10-17T23:50:25.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:25.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:26.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:26.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:27.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:27.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:28.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:28.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:29.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:29.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:29.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:29.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:29.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:29.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:30.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:50:30.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:30.114233+0000) 2019-10-17T23:50:30.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:30.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:30.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:30.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:31.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:31.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:31.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:31.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:31.616 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:50:31.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:31.613451+0000) 2019-10-17T23:50:32.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:32.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:32.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:32.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:32.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:50:32.545+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:32.550853+0000) 2019-10-17T23:50:33.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:33.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:33.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:33.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:34.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:34.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:34.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:34.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:34.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:34.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:34.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:50:34.585+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:34.592060+0000) 2019-10-17T23:50:35.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:35.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:35.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:35.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:36.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:36.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:36.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:36.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:37.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:37.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:37.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:37.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:38.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:38.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:38.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:38.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:39.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:39.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:39.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:39.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:39.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:39.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:40.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:50:40.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:40.114333+0000) 2019-10-17T23:50:40.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:40.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:40.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:40.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:41.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:41.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:41.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:41.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:41.617 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:50:41.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:41.613629+0000) 2019-10-17T23:50:42.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:42.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:42.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:42.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:42.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:50:42.549+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:42.550957+0000) 2019-10-17T23:50:43.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:43.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:43.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:43.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:44.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:44.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:44.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:44.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:44.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:44.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:44.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:50:44.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:44.592165+0000) 2019-10-17T23:50:45.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:45.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:45.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:45.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:46.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:46.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:46.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:46.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:47.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:47.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:47.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:47.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:48.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:48.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:48.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:48.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:49.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:49.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:49.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:49.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:49.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:49.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:50.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:50:50.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:50.114429+0000) 2019-10-17T23:50:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:50.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:50.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:51.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:51.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:51.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:51.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:51.617 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:50:51.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:51.613810+0000) 2019-10-17T23:50:52.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:52.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:52.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:52.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:52.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:50:52.549+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:52.551028+0000) 2019-10-17T23:50:53.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:53.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:53.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:53.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:54.403 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:54.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:54.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:54.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:54.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:54.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:54.595 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:50:54.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:50:54.592284+0000) 2019-10-17T23:50:55.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:55.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:55.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:55.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:56.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:56.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:56.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:56.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:57.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:57.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:57.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:57.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:58.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:58.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:58.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:58.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:50:59.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:50:59.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:50:59.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:59.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:50:59.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:50:59.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:00.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:51:00.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:00.114544+0000) 2019-10-17T23:51:00.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:00.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:00.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:00.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:01.617 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:51:01.608+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:01.613993+0000) 2019-10-17T23:51:01.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:01.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:01.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:01.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:02.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:51:02.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:02.551137+0000) 2019-10-17T23:51:02.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:02.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:02.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:02.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:03.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:03.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:03.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:03.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:04.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:04.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:04.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:51:04.588+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:04.592387+0000) 2019-10-17T23:51:04.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:04.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:04.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:04.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:05.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:05.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:05.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:05.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:06.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:06.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:06.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:06.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:07.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:07.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:07.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:07.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:08.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:08.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:08.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:08.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:09.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:09.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:09.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:09.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:09.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:09.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:10.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:51:10.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:10.114666+0000) 2019-10-17T23:51:10.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:10.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:10.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:10.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:11.617 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:51:11.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:11.614169+0000) 2019-10-17T23:51:11.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:11.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:11.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:11.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:12.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:51:12.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:12.551238+0000) 2019-10-17T23:51:12.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:12.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:12.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:12.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:13.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:13.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:13.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:13.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:14.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:14.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:14.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:51:14.588+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:14.592495+0000) 2019-10-17T23:51:14.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:14.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:14.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:14.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:15.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:15.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:15.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:16.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:16.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:16.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:16.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:17.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:17.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:17.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:17.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:18.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:18.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:18.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:18.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:19.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:19.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:19.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:19.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:19.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:19.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:20.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:51:20.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:20.114778+0000) 2019-10-17T23:51:20.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:20.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:20.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:20.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:21.617 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:51:21.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:21.614360+0000) 2019-10-17T23:51:21.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:21.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:21.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:21.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:22.557 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:51:22.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:22.551339+0000) 2019-10-17T23:51:22.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:22.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:22.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:22.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:23.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:23.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:23.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:23.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:24.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:24.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:24.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:51:24.588+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:24.592646+0000) 2019-10-17T23:51:24.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:24.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:24.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:24.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:25.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:25.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:25.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:25.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:26.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:26.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:27.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:27.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:27.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:27.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:28.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:28.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:28.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:28.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:29.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:29.404+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:29.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:29.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:29.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:29.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:30.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:51:30.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:30.114896+0000) 2019-10-17T23:51:30.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:30.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:30.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:30.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:31.617 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:51:31.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:31.614516+0000) 2019-10-17T23:51:31.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:31.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:31.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:31.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:32.554 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:51:32.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:32.551446+0000) 2019-10-17T23:51:32.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:32.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:32.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:32.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:33.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:33.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:33.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:33.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:34.410 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:34.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:34.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:51:34.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:34.592770+0000) 2019-10-17T23:51:34.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:34.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:34.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:34.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:35.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:35.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:35.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:35.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:36.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:36.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:36.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:36.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:37.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:37.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:37.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:37.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:38.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:38.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:38.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:38.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:39.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:39.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:39.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:39.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:39.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:39.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:40.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:51:40.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:40.115008+0000) 2019-10-17T23:51:40.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:40.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:40.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:40.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:41.618 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:51:41.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:41.614691+0000) 2019-10-17T23:51:41.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:41.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:41.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:41.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:42.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:51:42.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:42.551556+0000) 2019-10-17T23:51:42.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:42.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:42.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:42.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:43.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:43.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:43.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:43.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:44.412 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:44.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:44.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:51:44.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:44.592901+0000) 2019-10-17T23:51:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:44.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:44.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:45.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:45.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:45.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:45.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:46.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:46.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:46.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:46.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:47.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:47.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:47.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:47.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:48.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:48.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:48.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:48.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:49.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:49.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:49.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:49.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:49.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:49.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:50.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:51:50.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:50.115120+0000) 2019-10-17T23:51:50.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:50.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:50.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:50.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:51.618 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:51:51.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:51.614903+0000) 2019-10-17T23:51:51.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:51.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:51.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:51.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:52.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:51:52.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:52.551656+0000) 2019-10-17T23:51:52.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:52.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:52.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:52.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:53.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:53.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:53.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:53.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:54.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:54.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:54.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:51:54.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:51:54.593017+0000) 2019-10-17T23:51:54.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:54.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:54.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:54.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:55.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:55.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:55.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:55.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:56.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:56.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:56.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:56.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:57.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:57.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:57.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:57.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:58.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:58.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:58.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:58.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:51:59.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:51:59.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:51:59.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:59.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:51:59.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:51:59.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:00.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:52:00.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:00.115242+0000) 2019-10-17T23:52:00.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:00.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:00.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:00.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:01.618 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:52:01.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:01.615085+0000) 2019-10-17T23:52:01.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:01.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:01.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:01.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:02.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:52:02.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:02.551759+0000) 2019-10-17T23:52:02.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:02.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:02.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:02.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:03.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:03.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:03.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:03.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:04.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:04.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:04.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:52:04.587+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:04.593129+0000) 2019-10-17T23:52:04.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:04.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:04.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:04.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:05.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:05.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:05.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:05.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:06.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:06.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:06.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:06.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:07.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:07.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:07.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:07.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:08.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:08.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:08.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:08.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:09.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:09.406+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:09.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:09.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:09.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:09.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:10.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:52:10.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:10.115345+0000) 2019-10-17T23:52:10.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:10.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:10.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:10.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:11.618 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:52:11.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:11.615284+0000) 2019-10-17T23:52:11.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:11.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:11.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:11.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:12.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:52:12.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:12.551853+0000) 2019-10-17T23:52:12.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:12.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:12.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:12.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:13.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:13.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:13.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:13.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:14.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:14.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:14.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:52:14.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:14.593246+0000) 2019-10-17T23:52:14.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:14.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:14.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:14.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:15.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:15.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:16.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:16.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:17.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:17.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:17.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:17.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:18.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:18.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:18.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:18.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:19.418 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:19.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:19.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:19.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:19.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:19.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:20.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:52:20.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:20.115462+0000) 2019-10-17T23:52:20.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:20.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:20.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:20.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:21.618 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:52:21.611+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:21.615468+0000) 2019-10-17T23:52:21.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:21.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:21.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:21.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:22.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:52:22.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:22.551949+0000) 2019-10-17T23:52:22.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:22.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:22.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:22.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:23.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:23.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:23.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:23.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:24.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:24.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:24.596 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:52:24.586+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:24.593356+0000) 2019-10-17T23:52:25.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:24.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:25.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:24.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:26.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:26.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:27.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:27.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:27.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:27.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:28.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:28.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:28.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:28.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:29.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:29.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:29.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:29.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:29.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:29.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:30.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:52:30.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:30.115580+0000) 2019-10-17T23:52:30.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:30.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:30.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:30.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:31.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:31.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:31.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:31.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:31.619 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:52:31.610+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:31.615657+0000) 2019-10-17T23:52:32.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:32.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:32.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:32.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:32.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:52:32.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:32.552050+0000) 2019-10-17T23:52:33.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:33.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:33.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:33.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:34.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:34.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:34.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:34.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:34.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:34.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:34.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:52:34.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:34.593470+0000) 2019-10-17T23:52:35.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:35.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:35.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:35.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:36.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:36.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:36.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:36.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:37.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:37.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:37.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:37.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:38.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:38.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:38.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:38.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:39.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:39.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:39.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:39.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:40.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:52:40.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:40.115701+0000) 2019-10-17T23:52:40.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:40.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:40.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:40.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:41.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:41.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:41.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:41.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:41.619 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:52:41.610+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:41.615844+0000) 2019-10-17T23:52:42.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:42.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:42.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:42.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:42.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:52:42.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:42.552144+0000) 2019-10-17T23:52:43.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:43.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:43.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:43.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:44.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:44.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:44.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:44.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:44.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:44.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:44.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:52:44.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:44.593579+0000) 2019-10-17T23:52:45.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:45.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:45.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:45.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:46.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:46.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:46.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:46.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:47.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:47.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:47.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:47.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:48.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:48.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:48.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:48.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:49.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:49.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:49.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:49.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:49.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:49.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:50.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:52:50.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:50.115812+0000) 2019-10-17T23:52:50.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:50.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:50.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:50.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:51.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:51.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:51.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:51.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:51.620 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:52:51.610+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:51.615993+0000) 2019-10-17T23:52:52.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:52.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:52.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:52.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:52.560 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:52:52.545+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:52.552241+0000) 2019-10-17T23:52:53.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:53.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:53.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:53.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:54.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:54.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:54.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:54.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:54.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:54.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:52:54.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:52:54.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:52:54.593698+0000) 2019-10-17T23:52:55.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:55.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:55.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:55.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:56.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:56.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:56.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:56.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:57.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:57.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:57.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:57.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:58.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:58.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:58.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:58.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:59.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:59.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:52:59.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:52:59.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:52:59.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:52:59.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:00.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:53:00.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:00.115926+0000) 2019-10-17T23:53:00.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:00.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:00.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:00.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:01.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:01.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:01.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:01.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:01.619 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:53:01.610+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:01.616177+0000) 2019-10-17T23:53:02.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:02.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:02.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:02.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:02.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:53:02.549+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:02.552337+0000) 2019-10-17T23:53:03.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:03.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:03.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:03.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:04.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:04.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:04.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:04.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:04.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:04.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:04.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:53:04.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:04.593808+0000) 2019-10-17T23:53:05.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:05.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:05.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:05.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:06.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:06.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:06.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:06.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:07.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:07.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:07.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:07.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:08.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:08.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:08.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:08.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:09.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:09.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:09.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:09.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:09.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:09.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:10.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:53:10.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:10.116044+0000) 2019-10-17T23:53:10.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:10.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:10.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:10.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:11.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:11.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:11.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:11.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:11.619 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:53:11.610+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:11.616355+0000) 2019-10-17T23:53:12.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:12.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:12.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:12.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:12.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:53:12.549+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:12.552440+0000) 2019-10-17T23:53:13.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:13.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:13.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:13.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:14.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:14.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:14.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:14.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:14.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:14.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:14.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:53:14.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:14.593925+0000) 2019-10-17T23:53:15.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:15.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:15.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:15.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:16.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:16.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:16.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:16.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:17.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:17.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:17.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:17.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:18.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:18.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:18.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:18.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:19.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:19.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:19.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:19.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:19.427 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:19.420+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:20.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:53:20.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:20.116156+0000) 2019-10-17T23:53:20.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:20.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:20.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:20.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:21.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:21.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:21.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:21.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:21.619 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:53:21.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:21.616537+0000) 2019-10-17T23:53:22.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:22.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:22.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:22.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:22.555 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:53:22.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:22.552541+0000) 2019-10-17T23:53:23.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:23.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:23.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:23.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:24.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:24.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:24.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:24.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:24.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:24.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:24.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:53:24.592+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:24.594037+0000) 2019-10-17T23:53:25.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:25.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:25.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:25.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:26.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:26.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:26.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:26.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:27.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:27.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:27.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:27.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:28.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:28.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:29.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:29.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:29.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:29.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:29.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:29.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:30.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:53:30.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:30.116284+0000) 2019-10-17T23:53:30.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:30.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:30.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:30.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:31.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:31.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:31.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:31.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:31.620 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:53:31.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:31.616743+0000) 2019-10-17T23:53:32.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:32.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:32.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:32.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:32.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:53:32.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:32.552640+0000) 2019-10-17T23:53:33.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:33.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:33.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:33.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:34.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:34.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:34.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:34.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:34.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:34.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:34.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:53:34.592+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:34.594155+0000) 2019-10-17T23:53:35.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:35.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:35.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:35.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:36.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:36.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:36.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:36.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:37.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:37.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:37.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:37.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:38.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:38.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:38.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:38.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:39.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:39.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:39.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:39.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:39.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:39.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:40.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:53:40.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:40.116397+0000) 2019-10-17T23:53:40.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:40.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:40.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:40.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:41.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:41.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:41.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:41.620 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:53:41.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:41.616923+0000) 2019-10-17T23:53:42.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:42.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:42.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:42.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:42.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:53:42.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:42.552736+0000) 2019-10-17T23:53:43.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:43.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:43.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:43.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:44.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:44.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:44.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:44.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:44.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:44.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:44.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:53:44.592+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:44.594261+0000) 2019-10-17T23:53:45.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:45.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:45.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:45.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:46.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:46.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:46.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:46.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:47.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:47.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:47.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:47.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:48.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:48.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:48.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:48.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:49.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:49.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:49.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:49.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:49.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:49.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:50.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:53:50.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:50.116507+0000) 2019-10-17T23:53:50.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:50.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:50.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:50.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:51.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:51.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:51.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:51.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:51.620 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:53:51.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:51.617103+0000) 2019-10-17T23:53:52.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:52.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:52.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:52.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:52.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:53:52.548+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:52.552832+0000) 2019-10-17T23:53:53.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:53.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:53.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:53.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:54.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:54.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:54.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:54.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:54.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:54.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:53:54.597 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:53:54.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:53:54.594387+0000) 2019-10-17T23:53:55.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:55.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:55.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:55.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:56.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:56.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:56.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:56.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:57.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:57.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:57.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:57.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:58.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:58.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:58.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:58.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:59.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:59.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:53:59.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:53:59.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:53:59.434 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:53:59.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:00.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:54:00.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:00.116624+0000) 2019-10-17T23:54:00.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:00.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:00.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:00.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:01.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:01.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:01.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:01.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:01.620 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:54:01.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:01.617296+0000) 2019-10-17T23:54:02.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:02.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:02.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:02.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:02.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:02.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:02.552929+0000) 2019-10-17T23:54:03.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:03.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:03.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:03.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:04.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:04.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:04.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:04.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:04.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:04.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:04.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:54:04.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:04.594504+0000) 2019-10-17T23:54:05.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:05.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:05.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:05.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:06.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:06.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:06.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:06.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:07.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:07.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:07.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:07.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:08.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:08.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:08.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:08.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:09.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:09.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:09.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:09.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:09.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:09.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:10.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:54:10.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:10.116742+0000) 2019-10-17T23:54:10.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:10.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:10.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:10.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:11.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:11.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:11.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:11.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:11.620 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:54:11.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:11.617481+0000) 2019-10-17T23:54:12.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:12.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:12.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:12.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:12.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:12.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:12.553027+0000) 2019-10-17T23:54:13.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:13.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:13.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:13.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:14.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:14.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:14.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:14.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:14.427+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:14.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:54:14.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:14.594624+0000) 2019-10-17T23:54:15.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:15.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:15.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:15.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:16.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:16.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:16.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:16.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:17.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:17.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:17.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:17.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:18.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:18.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:18.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:18.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:19.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:19.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:19.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:19.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:19.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:19.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:20.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:54:20.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:20.116854+0000) 2019-10-17T23:54:20.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:20.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:20.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:20.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:21.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:21.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:21.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:21.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:21.621 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:54:21.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:21.617660+0000) 2019-10-17T23:54:22.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:22.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:22.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:22.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:22.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:22.547+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:22.553125+0000) 2019-10-17T23:54:23.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:23.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:23.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:23.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:24.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:24.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:24.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:24.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:24.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:24.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:24.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:54:24.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:24.594733+0000) 2019-10-17T23:54:25.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:25.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:25.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:25.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:26.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:26.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:26.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:26.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:27.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:27.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:27.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:27.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:28.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:28.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:28.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:28.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:29.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:29.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:29.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:29.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:29.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:29.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:30.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:54:30.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:30.116973+0000) 2019-10-17T23:54:30.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:30.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:30.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:30.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:31.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:31.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:31.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:31.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:31.621 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:54:31.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:31.617867+0000) 2019-10-17T23:54:32.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:32.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:32.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:32.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:32.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:32.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:32.553231+0000) 2019-10-17T23:54:33.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:33.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:33.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:33.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:34.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:34.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:34.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:34.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:34.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:34.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:34.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:54:34.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:34.594857+0000) 2019-10-17T23:54:35.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:35.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:35.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:35.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:36.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:36.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:36.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:36.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:37.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:37.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:37.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:37.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:38.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:38.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:38.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:38.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:39.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:39.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:39.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:39.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:39.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:39.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:40.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:54:40.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:40.117092+0000) 2019-10-17T23:54:40.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:40.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:40.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:40.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:41.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:41.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:41.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:41.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:41.621 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:54:41.613+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:41.618054+0000) 2019-10-17T23:54:42.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:42.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:42.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:42.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:42.556 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:42.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:42.553325+0000) 2019-10-17T23:54:43.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:43.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:43.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:43.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:44.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:44.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:44.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:44.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:44.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:44.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:44.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:54:44.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:44.594966+0000) 2019-10-17T23:54:45.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:45.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:45.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:45.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:46.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:46.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:46.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:46.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:47.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:47.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:47.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:47.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:48.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:48.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:48.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:48.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:49.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:49.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:49.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:49.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:49.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:49.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:50.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:54:50.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:50.117204+0000) 2019-10-17T23:54:50.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:50.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:50.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:50.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:51.621 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:54:51.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:51.618259+0000) 2019-10-17T23:54:51.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:51.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:51.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:51.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:52.557 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:52.546+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:52.553419+0000) 2019-10-17T23:54:52.557 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:52.550+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:52.554639+0000) 2019-10-17T23:54:52.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:52.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:52.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:52.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:53.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:53.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:53.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:53.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:54.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:54.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:54.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:54.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:54.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:54.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:54.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:54:54.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:54.595085+0000) 2019-10-17T23:54:55.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:55.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:55.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:55.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:56.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:56.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:56.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:56.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:57.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:54:57.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:54:57.053639+0000) 2019-10-17T23:54:57.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:57.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:57.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:57.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:58.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:58.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:54:59.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:54:59.438+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:54:59.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:59.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:54:59.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:54:59.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:00.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:55:00.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:00.117329+0000) 2019-10-17T23:55:00.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:00.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:00.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:00.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:01.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:01.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:01.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:01.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:01.621 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:55:01.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:01.618406+0000) 2019-10-17T23:55:02.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:02.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:02.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:02.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:03.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:03.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:03.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:03.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:04.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:04.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:04.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:04.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:04.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:04.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:04.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:55:04.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:04.595199+0000) 2019-10-17T23:55:05.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:05.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:05.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:05.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:06.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:06.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:06.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:06.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:07.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:55:07.049+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:07.053741+0000) 2019-10-17T23:55:07.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:07.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:07.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:07.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:08.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:08.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:08.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:08.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:09.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:09.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:09.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:09.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:09.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:09.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:10.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:55:10.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:10.117451+0000) 2019-10-17T23:55:10.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:10.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:10.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:10.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:11.622 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:55:11.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:11.618587+0000) 2019-10-17T23:55:11.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:11.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:11.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:11.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:12.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:12.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:12.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:12.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:13.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:13.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:13.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:13.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:14.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:14.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:14.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:55:14.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:14.595323+0000) 2019-10-17T23:55:14.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:14.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:14.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:14.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:15.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:15.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:15.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:15.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:16.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:16.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:16.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:16.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:17.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:55:17.049+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:17.053843+0000) 2019-10-17T23:55:17.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:17.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:17.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:17.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:18.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:18.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:18.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:18.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:19.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:19.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:19.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:19.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:19.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:19.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:20.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:55:20.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:20.117566+0000) 2019-10-17T23:55:20.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:20.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:20.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:20.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:21.622 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:55:21.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:21.618785+0000) 2019-10-17T23:55:21.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:21.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:21.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:21.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:22.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:22.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:22.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:22.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:23.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:23.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:23.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:23.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:24.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:24.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:24.598 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:55:24.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:24.595434+0000) 2019-10-17T23:55:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:24.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:24.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:25.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:25.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:26.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:26.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:26.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:26.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:27.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:55:27.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:27.053950+0000) 2019-10-17T23:55:27.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:27.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:27.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:27.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:28.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:28.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:28.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:28.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:29.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:29.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:29.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:29.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:29.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:29.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:30.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:55:30.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:30.117685+0000) 2019-10-17T23:55:30.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:30.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:30.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:30.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:31.622 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:55:31.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:31.618964+0000) 2019-10-17T23:55:31.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:31.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:31.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:31.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:32.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:32.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:32.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:32.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:33.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:33.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:33.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:33.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:34.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:34.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:34.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:55:34.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:34.595559+0000) 2019-10-17T23:55:34.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:34.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:34.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:34.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:35.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:35.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:35.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:35.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:36.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:36.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:36.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:36.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:37.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:55:37.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:37.054051+0000) 2019-10-17T23:55:37.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:37.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:37.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:37.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:38.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:38.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:38.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:38.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:39.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:39.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:39.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:39.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:39.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:39.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:40.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:55:40.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:40.117800+0000) 2019-10-17T23:55:40.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:40.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:40.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:40.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:41.624 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:55:41.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:41.619144+0000) 2019-10-17T23:55:41.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:41.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:41.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:41.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:42.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:42.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:42.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:42.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:43.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:43.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:43.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:43.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:44.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:44.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:44.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:55:44.592+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:44.595688+0000) 2019-10-17T23:55:44.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:44.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:44.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:44.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:45.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:45.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:45.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:45.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:46.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:46.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:46.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:46.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:47.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:55:47.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:47.054155+0000) 2019-10-17T23:55:47.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:47.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:47.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:47.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:48.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:48.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:48.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:48.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:49.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:49.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:49.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:49.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:49.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:49.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:50.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:55:50.112+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:50.117897+0000) 2019-10-17T23:55:50.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:50.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:50.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:50.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:51.622 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:55:51.616+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:51.619352+0000) 2019-10-17T23:55:51.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:51.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:51.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:51.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:52.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:52.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:52.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:52.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:53.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:53.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:53.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:53.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:54.454 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:54.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:54.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:55:54.592+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:54.595805+0000) 2019-10-17T23:55:54.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:54.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:54.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:54.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:55.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:55.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:55.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:55.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:57.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:56.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:57.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:56.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:57.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:55:57.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:55:57.054256+0000) 2019-10-17T23:55:57.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:57.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:57.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:57.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:58.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:58.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:58.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:58.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:55:59.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:55:59.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:55:59.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:59.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:55:59.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:55:59.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:00.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:56:00.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:00.118018+0000) 2019-10-17T23:56:00.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:00.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:00.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:00.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:01.622 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:56:01.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:01.619550+0000) 2019-10-17T23:56:01.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:01.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:01.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:01.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:02.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:02.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:02.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:02.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:03.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:03.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:03.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:03.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:04.455 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:04.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:04.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:56:04.592+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:04.595911+0000) 2019-10-17T23:56:04.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:04.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:04.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:04.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:05.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:05.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:05.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:05.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:06.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:06.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:06.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:06.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:07.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:56:07.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:07.054357+0000) 2019-10-17T23:56:07.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:07.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:07.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:07.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:08.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:08.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:08.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:08.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:09.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:09.452+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:09.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:09.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:09.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:09.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:10.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:56:10.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:10.118132+0000) 2019-10-17T23:56:10.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:10.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:10.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:10.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:11.623 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:56:11.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:11.619761+0000) 2019-10-17T23:56:11.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:11.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:11.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:11.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:12.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:12.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:12.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:12.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:13.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:13.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:13.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:13.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:14.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:14.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:14.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:56:14.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:14.596031+0000) 2019-10-17T23:56:14.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:14.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:14.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:14.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:15.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:15.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:15.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:15.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:16.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:16.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:16.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:16.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:17.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:56:17.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:17.054459+0000) 2019-10-17T23:56:17.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:17.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:17.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:17.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:18.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:18.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:18.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:18.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:19.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:19.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:19.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:19.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:19.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:19.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:20.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:56:20.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:20.118256+0000) 2019-10-17T23:56:20.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:20.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:20.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:20.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:21.623 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:56:21.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:21.619977+0000) 2019-10-17T23:56:21.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:21.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:21.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:21.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:22.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:22.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:22.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:22.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:23.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:23.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:23.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:23.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:24.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:24.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:24.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:56:24.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:24.596143+0000) 2019-10-17T23:56:24.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:24.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:24.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:24.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:25.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:25.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:25.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:25.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:26.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:26.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:26.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:26.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:27.057 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:56:27.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:27.054559+0000) 2019-10-17T23:56:27.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:27.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:27.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:27.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:28.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:28.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:28.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:28.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:29.460 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:29.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:29.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:29.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:29.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:29.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:30.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:56:30.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:30.118362+0000) 2019-10-17T23:56:30.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:30.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:30.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:30.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:31.623 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:56:31.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:31.620182+0000) 2019-10-17T23:56:31.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:31.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:31.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:31.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:32.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:32.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:32.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:32.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:33.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:33.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:33.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:33.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:34.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:34.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:34.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:56:34.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:34.596274+0000) 2019-10-17T23:56:34.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:34.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:34.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:34.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:35.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:35.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:35.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:35.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:36.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:36.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:36.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:36.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:37.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:56:37.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:37.054658+0000) 2019-10-17T23:56:38.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:38.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:38.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:38.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:39.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:39.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:39.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:39.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:39.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:39.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:40.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:40.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:40.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:56:40.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:40.118463+0000) 2019-10-17T23:56:41.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:41.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:41.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:41.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:41.623 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:56:41.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:41.620333+0000) 2019-10-17T23:56:42.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:42.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:42.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:42.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:43.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:42.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:43.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:42.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:44.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:44.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:44.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:44.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:44.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:44.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:44.599 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:56:44.591+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:44.596388+0000) 2019-10-17T23:56:45.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:45.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:45.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:45.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:46.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:46.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:46.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:46.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:47.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:56:47.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:47.054759+0000) 2019-10-17T23:56:47.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:47.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:47.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:48.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:48.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:48.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:48.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:49.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:49.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:49.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:49.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:49.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:49.454+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:50.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:50.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:50.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:56:50.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:50.118565+0000) 2019-10-17T23:56:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:51.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:51.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:51.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:51.623 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:56:51.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:51.620542+0000) 2019-10-17T23:56:52.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:52.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:52.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:52.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:53.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:53.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:53.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:53.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:54.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:54.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:54.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:54.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:54.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:54.454+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:56:54.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:56:54.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:54.596517+0000) 2019-10-17T23:56:55.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:55.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:55.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:55.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:56.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:56.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:56.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:56.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:57.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:56:57.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:56:57.054855+0000) 2019-10-17T23:56:57.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:57.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:57.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:57.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:58.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:58.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:58.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:58.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:59.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:59.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:56:59.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:56:59.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:56:59.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:56:59.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:00.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:57:00.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:00.118680+0000) 2019-10-17T23:57:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:00.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:00.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:01.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:01.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:01.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:01.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:01.624 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:57:01.615+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:01.620683+0000) 2019-10-17T23:57:02.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:02.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:02.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:02.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:03.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:03.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:03.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:03.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:04.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:04.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:04.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:04.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:04.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:04.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:04.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:57:04.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:04.596627+0000) 2019-10-17T23:57:05.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:05.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:05.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:05.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:06.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:06.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:06.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:06.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:07.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:57:07.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:07.054956+0000) 2019-10-17T23:57:07.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:07.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:07.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:07.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:08.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:08.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:08.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:08.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:09.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:09.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:09.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:09.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:09.467 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:09.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:10.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:57:10.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:10.118774+0000) 2019-10-17T23:57:10.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:10.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:10.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:10.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:11.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:11.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:11.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:11.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:11.624 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:57:11.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:11.620888+0000) 2019-10-17T23:57:12.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:12.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:12.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:12.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:13.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:13.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:13.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:13.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:14.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:14.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:14.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:14.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:14.467 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:14.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:14.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:57:14.590+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:14.596750+0000) 2019-10-17T23:57:15.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:15.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:15.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:15.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:16.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:16.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:16.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:16.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:17.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:17.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:17.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:17.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:17.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:57:17.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:17.055056+0000) 2019-10-17T23:57:18.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:18.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:18.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:18.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:19.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:19.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:19.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:19.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:19.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:19.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:20.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:20.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:20.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:20.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:20.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:57:20.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:20.118884+0000) 2019-10-17T23:57:21.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:21.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:21.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:21.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:21.624 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:57:21.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:21.621099+0000) 2019-10-17T23:57:22.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:22.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:22.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:22.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:23.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:23.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:23.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:23.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:24.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:24.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:24.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:24.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:24.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:24.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:24.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:57:24.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:24.596863+0000) 2019-10-17T23:57:25.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:25.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:25.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:25.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:26.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:26.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:26.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:26.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:27.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:27.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:27.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:27.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:27.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:57:27.049+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:27.055158+0000) 2019-10-17T23:57:28.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:28.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:28.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:28.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:29.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:29.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:29.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:29.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:29.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:29.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:30.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:30.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:30.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:30.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:30.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:57:30.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:30.118996+0000) 2019-10-17T23:57:31.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:31.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:31.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:31.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:31.624 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:57:31.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:31.621301+0000) 2019-10-17T23:57:32.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:32.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:32.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:32.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:33.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:33.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:33.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:33.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:34.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:34.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:34.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:34.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:34.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:34.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:34.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:57:34.589+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:34.596985+0000) 2019-10-17T23:57:35.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:35.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:35.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:35.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:36.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:36.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:36.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:36.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:37.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:37.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:37.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:37.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:37.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:57:37.049+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:37.055260+0000) 2019-10-17T23:57:37.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:37.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:37.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:37.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:38.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:38.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:38.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:38.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:39.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:39.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:39.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:39.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:39.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:39.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:40.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:57:40.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:40.119111+0000) 2019-10-17T23:57:40.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:40.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:40.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:40.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:41.624 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:57:41.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:41.621509+0000) 2019-10-17T23:57:41.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:41.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:41.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:41.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:42.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:42.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:42.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:42.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:43.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:43.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:44.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:43.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:44.472 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:44.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:44.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:57:44.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:44.597104+0000) 2019-10-17T23:57:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:45.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:45.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:46.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:46.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:46.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:46.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:47.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:46.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:47.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:46.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:47.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:57:47.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:47.055361+0000) 2019-10-17T23:57:47.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:47.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:47.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:47.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:48.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:48.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:48.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:48.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:49.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:49.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:49.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:49.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:49.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:49.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:50.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:57:50.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:50.119223+0000) 2019-10-17T23:57:50.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:50.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:50.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:50.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:51.629 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:57:51.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:51.621685+0000) 2019-10-17T23:57:51.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:51.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:51.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:51.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:52.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:52.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:52.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:52.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:53.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:53.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:53.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:53.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:54.474 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:54.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:54.600 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:57:54.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:54.597239+0000) 2019-10-17T23:57:54.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:54.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:54.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:54.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:55.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:55.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:55.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:55.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:56.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:56.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:56.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:56.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:57.058 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:57:57.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:57:57.055463+0000) 2019-10-17T23:57:57.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:57.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:57.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:57.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:58.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:58.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:58.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:58.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:57:59.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:57:59.468+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:57:59.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:59.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:57:59.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:57:59.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:00.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:58:00.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:00.119338+0000) 2019-10-17T23:58:00.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:00.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:00.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:00.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:01.625 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:58:01.614+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:01.621877+0000) 2019-10-17T23:58:01.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:01.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:01.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:01.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:02.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:02.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:02.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:02.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:03.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:03.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:04.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:04.468+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:04.603 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:58:04.596+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:04.600006+0000) 2019-10-17T23:58:04.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:04.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:04.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:04.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:05.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:05.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:05.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:05.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:06.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:06.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:06.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:06.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:07.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:58:07.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:07.055568+0000) 2019-10-17T23:58:07.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:07.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:07.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:07.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:08.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:08.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:08.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:08.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:09.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:09.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:09.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:09.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:09.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:09.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:10.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:58:10.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:10.119450+0000) 2019-10-17T23:58:10.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:10.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:10.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:10.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:11.625 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:58:11.618+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:11.622060+0000) 2019-10-17T23:58:11.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:11.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:11.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:11.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:12.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:12.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:12.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:12.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:13.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:13.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:14.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:14.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:14.603 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:58:14.596+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:14.600124+0000) 2019-10-17T23:58:14.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:14.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:14.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:14.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:15.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:15.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:15.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:15.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:16.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:16.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:16.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:16.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:17.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:58:17.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:17.055670+0000) 2019-10-17T23:58:17.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:17.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:17.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:17.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:18.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:18.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:19.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:19.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:19.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:19.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:19.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:19.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:20.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:58:20.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:20.119579+0000) 2019-10-17T23:58:20.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:20.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:20.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:20.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:21.625 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:58:21.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:21.622259+0000) 2019-10-17T23:58:21.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:21.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:21.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:21.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:22.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:22.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:23.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:23.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:23.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:23.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:24.479 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:24.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:24.603 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:58:24.596+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:24.600233+0000) 2019-10-17T23:58:24.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:24.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:24.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:24.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:25.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:25.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:25.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:25.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:26.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:26.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:26.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:26.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:27.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:58:27.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:27.055769+0000) 2019-10-17T23:58:27.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:27.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:27.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:27.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:28.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:28.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:28.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:28.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:29.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:29.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:29.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:29.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:29.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:29.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:30.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:58:30.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:30.119692+0000) 2019-10-17T23:58:30.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:30.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:30.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:30.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:31.627 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:58:31.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:31.622406+0000) 2019-10-17T23:58:31.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:31.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:31.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:31.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:32.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:32.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:32.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:32.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:33.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:33.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:33.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:33.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:34.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:34.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:34.603 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:58:34.595+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:34.600364+0000) 2019-10-17T23:58:34.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:34.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:34.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:34.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:35.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:35.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:35.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:35.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:36.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:36.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:36.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:36.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:37.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:58:37.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:37.055871+0000) 2019-10-17T23:58:37.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:37.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:37.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:37.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:38.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:38.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:38.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:38.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:39.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:39.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:39.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:39.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:39.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:39.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:40.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:58:40.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:40.119810+0000) 2019-10-17T23:58:40.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:40.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:40.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:40.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:41.625 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:58:41.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:41.622550+0000) 2019-10-17T23:58:41.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:41.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:41.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:41.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:42.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:42.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:42.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:42.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:43.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:43.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:43.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:43.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:44.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:44.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:44.606 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:58:44.595+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:44.600475+0000) 2019-10-17T23:58:44.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:44.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:44.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:44.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:45.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:45.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:45.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:45.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:46.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:46.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:46.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:46.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:47.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:58:47.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:47.055974+0000) 2019-10-17T23:58:47.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:47.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:47.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:47.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:48.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:48.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:48.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:48.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:49.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:49.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:49.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:49.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:49.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:49.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:50.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:58:50.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:50.119925+0000) 2019-10-17T23:58:50.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:50.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:50.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:50.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:51.626 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:58:51.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:51.622716+0000) 2019-10-17T23:58:51.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:51.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:51.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:51.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:52.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:52.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:52.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:52.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:53.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:53.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:53.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:53.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:54.486 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:54.475+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:54.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:58:54.595+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:54.600611+0000) 2019-10-17T23:58:54.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:54.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:54.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:54.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:55.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:55.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:55.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:55.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:56.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:56.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:56.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:56.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:57.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:58:57.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:58:57.056077+0000) 2019-10-17T23:58:57.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:57.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:57.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:57.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:58.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:58.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:58.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:58.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:58:59.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:58:59.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:58:59.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:59.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:58:59.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:58:59.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:00.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:59:00.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:00.120057+0000) 2019-10-17T23:59:00.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:00.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:00.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:00.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:01.626 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:59:01.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:01.622905+0000) 2019-10-17T23:59:01.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:01.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:01.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:01.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:02.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:02.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:02.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:02.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:03.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:03.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:03.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:03.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:04.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:04.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:04.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:59:04.595+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:04.600725+0000) 2019-10-17T23:59:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:04.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:04.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:04.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:05.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:05.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:05.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:05.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:06.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:06.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:06.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:06.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:07.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:59:07.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:07.056181+0000) 2019-10-17T23:59:07.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:07.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:07.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:07.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:08.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:08.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:08.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:08.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:09.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:09.478+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:09.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:09.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:09.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:09.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:10.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:59:10.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:10.120176+0000) 2019-10-17T23:59:10.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:10.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:10.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:10.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:11.626 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:59:11.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:11.623086+0000) 2019-10-17T23:59:11.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:11.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:11.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:11.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:12.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:12.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:12.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:12.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:13.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:13.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:13.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:13.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:14.488 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:14.478+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:14.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:59:14.594+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:14.600857+0000) 2019-10-17T23:59:14.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:14.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:14.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:14.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:15.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:15.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:15.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:15.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:16.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:16.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:16.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:16.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:17.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:59:17.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:17.056282+0000) 2019-10-17T23:59:17.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:17.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:17.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:17.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:18.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:18.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:18.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:18.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:19.489 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:19.478+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:19.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:19.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:19.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:19.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:20.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:59:20.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:20.120285+0000) 2019-10-17T23:59:20.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:20.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:20.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:20.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:21.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:21.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:21.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:21.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:21.626 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:59:21.621+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:21.623267+0000) 2019-10-17T23:59:22.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:22.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:22.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:22.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:23.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:23.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:23.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:23.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:24.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:24.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:24.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:24.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:24.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:24.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:24.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:59:24.594+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:24.600973+0000) 2019-10-17T23:59:25.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:25.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:25.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:25.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:26.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:26.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:26.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:26.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:27.059 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:59:27.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:27.056390+0000) 2019-10-17T23:59:27.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:27.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:27.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:27.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:28.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:28.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:28.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:28.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:29.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:29.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:29.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:29.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:29.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:29.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:30.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:59:30.117+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:30.120395+0000) 2019-10-17T23:59:30.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:30.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:30.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:30.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:31.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:31.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:31.626 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:59:31.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:31.623471+0000) 2019-10-17T23:59:32.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:32.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:32.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:32.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:33.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:33.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:33.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:33.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:34.491 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:34.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:34.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:59:34.594+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:34.601100+0000) 2019-10-17T23:59:34.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:34.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:34.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:34.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:35.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:35.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:35.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:35.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:36.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:36.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:36.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:36.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:37.063 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:59:37.050+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:37.056492+0000) 2019-10-17T23:59:37.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:37.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:37.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:37.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:38.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:38.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:38.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:38.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:39.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:39.482+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:39.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:39.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:39.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:39.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:40.123 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:59:40.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:40.120516+0000) 2019-10-17T23:59:40.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:40.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:40.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:40.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:41.627 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:59:41.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:41.623652+0000) 2019-10-17T23:59:41.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:41.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:41.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:41.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:42.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:42.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:42.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:42.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:43.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:43.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:43.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:43.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:44.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:44.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:44.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:59:44.594+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:44.601215+0000) 2019-10-17T23:59:44.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:44.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:44.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:45.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:45.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:45.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:45.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:46.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:46.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:46.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:46.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:47.069 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:59:47.049+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:47.056599+0000) 2019-10-17T23:59:47.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:47.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:47.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:47.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:48.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:48.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:48.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:48.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:49.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:49.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:49.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:49.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:49.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:49.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:50.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-17T23:59:50.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:50.120649+0000) 2019-10-17T23:59:50.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:50.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:50.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:50.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:51.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:51.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:51.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:51.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:51.627 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-17T23:59:51.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:51.623843+0000) 2019-10-17T23:59:52.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:52.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:52.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:52.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:53.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:53.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:53.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:53.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:54.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:54.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:54.604 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-17T23:59:54.593+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:54.601338+0000) 2019-10-17T23:59:54.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:54.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:54.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:54.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:55.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:55.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:55.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:55.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:56.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:56.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:56.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:56.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:57.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-17T23:59:57.049+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T22:59:57.056699+0000) 2019-10-17T23:59:57.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:57.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:57.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:57.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:58.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:58.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:58.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:58.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-17T23:59:59.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-17T23:59:59.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-17T23:59:59.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:59.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-17T23:59:59.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-17T23:59:59.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:00.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:00:00.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:00.120775+0000) 2019-10-18T00:00:00.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:00.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:00.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:00.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:01.627 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:00:01.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:01.624031+0000) 2019-10-18T00:00:01.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:01.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:01.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:01.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:02.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:02.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:02.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:02.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:03.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:03.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:03.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:03.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:04.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:04.489+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:04.606 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:04.601+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:04.601460+0000) 2019-10-18T00:00:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:04.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:04.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:05.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:05.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:05.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:05.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:06.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:06.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:06.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:06.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:07.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:00:07.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:07.056802+0000) 2019-10-18T00:00:07.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:07.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:07.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:07.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:08.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:08.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:08.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:08.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:09.497 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:09.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:09.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:09.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:09.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:09.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:10.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:00:10.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:10.120906+0000) 2019-10-18T00:00:10.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:10.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:10.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:10.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:11.627 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:00:11.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:11.624210+0000) 2019-10-18T00:00:11.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:11.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:11.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:11.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:12.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:12.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:12.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:13.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:13.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:13.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:13.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:14.498 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:14.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:14.605 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:14.601+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:14.601603+0000) 2019-10-18T00:00:14.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:14.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:14.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:14.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:15.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:15.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:15.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:15.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:16.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:16.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:16.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:16.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:17.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:00:17.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:17.056904+0000) 2019-10-18T00:00:17.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:17.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:17.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:17.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:18.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:18.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:18.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:18.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:19.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:19.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:19.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:19.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:19.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:19.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:20.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:00:20.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:20.121045+0000) 2019-10-18T00:00:20.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:20.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:20.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:20.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:21.627 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:00:21.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:21.624391+0000) 2019-10-18T00:00:21.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:21.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:21.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:21.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:22.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:22.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:22.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:22.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:23.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:23.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:23.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:23.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:24.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:24.492+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:24.605 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:24.600+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:24.601718+0000) 2019-10-18T00:00:24.606 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:24.600+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:24.602848+0000) 2019-10-18T00:00:24.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:24.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:24.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:24.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:25.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:25.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:25.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:25.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:26.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:26.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:26.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:26.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:27.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:00:27.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:27.057013+0000) 2019-10-18T00:00:27.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:27.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:27.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:27.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:28.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:28.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:28.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:28.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:29.105 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:29.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:29.101969+0000) 2019-10-18T00:00:29.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:29.492+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:30.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:30.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:30.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:30.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:30.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:00:30.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:30.121160+0000) 2019-10-18T00:00:31.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:31.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:31.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:31.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:31.628 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:00:31.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:31.624571+0000) 2019-10-18T00:00:31.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:31.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:31.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:31.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:32.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:32.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:32.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:32.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:33.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:33.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:33.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:33.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:34.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:34.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:34.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:34.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:34.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:34.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:35.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:35.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:35.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:35.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:36.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:36.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:36.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:36.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:37.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:00:37.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:37.057115+0000) 2019-10-18T00:00:37.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:37.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:37.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:37.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:38.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:38.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-17T23:50:09.139118+0000 (oldest deadline 2019-10-17T23:50:29.139118+0000) 2019-10-18T00:00:38.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:38.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:39.105 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:39.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:39.102071+0000) 2019-10-18T00:00:39.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:39.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:39.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:39.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:40.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:00:40.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:40.121278+0000) 2019-10-18T00:00:40.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:40.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:41.628 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:00:41.620+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:41.624759+0000) 2019-10-18T00:00:41.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:41.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:42.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:42.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:43.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:43.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:44.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:44.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:44.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:44.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:45.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:45.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:46.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:46.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:47.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:00:47.052+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:47.057215+0000) 2019-10-18T00:00:47.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:47.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:48.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:48.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:49.105 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:49.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:49.102170+0000) 2019-10-18T00:00:49.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:49.496+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:49.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:49.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:50.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:00:50.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:50.121406+0000) 2019-10-18T00:00:50.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:50.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:51.628 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:00:51.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:51.624943+0000) 2019-10-18T00:00:51.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:51.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:52.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:52.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:53.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:53.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:54.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:54.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:54.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:54.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:55.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:55.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:56.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:56.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:57.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:00:57.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:57.057314+0000) 2019-10-18T00:00:57.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:57.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:58.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:58.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:00:59.105 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:00:59.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:00:59.102266+0000) 2019-10-18T00:00:59.506 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:00:59.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:00:59.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:00:59.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:00.124 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:01:00.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:00.121537+0000) 2019-10-18T00:01:00.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:00.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:01.628 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:01:01.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:01.625128+0000) 2019-10-18T00:01:01.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:01.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:01.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:01.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:02.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:02.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:02.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:02.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:03.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:03.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:03.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:03.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:04.506 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:04.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:04.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:04.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:04.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:04.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:05.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:05.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:05.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:05.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:06.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:06.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:06.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:06.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:07.060 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:01:07.051+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:07.057415+0000) 2019-10-18T00:01:07.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:07.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:07.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:07.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:08.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:08.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:08.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:08.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:09.105 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:01:09.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:09.102366+0000) 2019-10-18T00:01:09.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:09.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:09.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:09.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:09.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:09.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:10.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:01:10.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:10.121659+0000) 2019-10-18T00:01:10.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:10.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:10.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:10.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:11.628 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:01:11.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:11.625334+0000) 2019-10-18T00:01:11.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:11.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:11.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:11.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:12.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:12.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:12.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:12.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:13.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:13.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:13.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:13.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:14.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:14.499+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:14.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:14.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:14.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:14.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:15.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:15.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:15.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:15.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:16.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:16.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:16.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:16.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:17.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:01:17.055+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:17.057521+0000) 2019-10-18T00:01:17.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:17.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:17.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:17.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:18.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:18.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:19.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:01:19.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:19.102469+0000) 2019-10-18T00:01:19.510 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:19.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:19.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:19.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:19.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:19.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:20.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:01:20.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:20.121778+0000) 2019-10-18T00:01:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:20.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:20.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:21.628 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:01:21.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:21.625518+0000) 2019-10-18T00:01:21.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:21.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:21.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:21.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:22.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:22.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:22.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:22.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:23.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:23.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:23.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:23.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:24.510 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:24.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:24.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:24.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:24.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:24.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:25.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:25.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:25.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:25.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:26.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:26.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:26.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:26.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:27.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:01:27.055+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:27.057625+0000) 2019-10-18T00:01:27.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:27.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:27.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:27.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:28.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:28.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:28.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:28.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:29.105 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:01:29.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:29.102569+0000) 2019-10-18T00:01:29.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:29.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:29.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:29.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:29.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:29.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:30.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:01:30.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:30.121894+0000) 2019-10-18T00:01:30.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:30.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:30.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:30.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:31.629 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:01:31.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:31.625695+0000) 2019-10-18T00:01:31.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:31.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:31.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:31.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:32.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:32.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:32.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:32.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:33.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:33.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:33.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:33.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:34.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:34.502+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:34.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:34.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:34.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:34.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:35.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:35.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:35.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:35.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:36.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:36.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:36.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:36.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:37.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:01:37.054+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:37.057727+0000) 2019-10-18T00:01:37.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:37.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:37.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:37.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:38.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:38.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:38.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:38.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:39.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:01:39.098+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:39.102668+0000) 2019-10-18T00:01:39.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:39.502+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:39.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:39.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:39.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:39.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:40.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:01:40.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:40.122007+0000) 2019-10-18T00:01:40.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:40.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:40.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:40.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:41.629 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:01:41.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:41.625874+0000) 2019-10-18T00:01:41.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:41.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:41.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:41.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:42.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:42.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:42.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:42.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:43.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:43.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:43.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:43.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:44.513 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:44.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:44.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:44.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:45.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:45.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:45.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:45.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:46.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:46.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:46.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:46.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:47.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:01:47.054+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:47.057831+0000) 2019-10-18T00:01:47.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:47.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:47.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:47.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:48.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:48.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:48.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:48.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:49.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:01:49.098+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:49.102768+0000) 2019-10-18T00:01:49.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:49.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:49.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:49.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:49.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:49.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:50.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:01:50.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:50.122125+0000) 2019-10-18T00:01:51.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:51.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:51.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:51.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:51.629 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:01:51.619+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:51.626060+0000) 2019-10-18T00:01:51.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:51.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:51.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:51.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:53.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:52.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:53.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:52.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:54.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:54.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:54.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:54.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:54.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:54.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:01:55.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:55.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:55.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:55.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:56.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:56.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:56.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:56.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:57.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:56.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:57.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:56.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:57.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:01:57.054+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:57.057933+0000) 2019-10-18T00:01:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:58.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:58.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:59.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:59.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:01:59.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:01:59.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:01:59.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:01:59.098+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:01:59.102862+0000) 2019-10-18T00:01:59.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:01:59.506+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:00.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:00.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:00.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:00.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:00.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:02:00.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:00.122262+0000) 2019-10-18T00:02:01.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:01.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:01.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:01.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:01.633 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:02:01.622+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:01.626263+0000) 2019-10-18T00:02:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:02.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:02.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:03.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:03.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:03.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:03.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:04.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:04.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:04.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:04.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:04.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:04.510+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:05.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:05.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:05.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:05.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:06.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:06.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:06.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:06.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:07.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:02:07.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:07.058034+0000) 2019-10-18T00:02:07.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:07.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:07.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:07.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:08.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:08.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:08.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:08.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:09.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:02:09.097+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:09.102962+0000) 2019-10-18T00:02:09.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:09.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:09.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:09.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:09.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:09.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:10.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:02:10.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:10.122361+0000) 2019-10-18T00:02:10.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:10.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:10.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:10.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:11.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:11.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:11.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:11.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:11.629 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:02:11.622+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:11.626408+0000) 2019-10-18T00:02:12.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:12.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:12.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:12.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:13.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:13.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:13.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:13.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:14.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:14.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:14.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:14.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:14.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:14.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:15.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:15.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:15.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:15.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:16.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:16.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:16.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:16.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:17.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:02:17.053+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:17.058132+0000) 2019-10-18T00:02:17.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:17.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:17.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:17.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:18.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:18.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:18.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:18.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:19.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:19.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:19.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:19.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:19.107 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:02:19.097+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:19.103071+0000) 2019-10-18T00:02:19.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:19.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:20.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:20.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:20.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:20.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:20.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:02:20.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:20.122457+0000) 2019-10-18T00:02:21.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:21.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:21.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:21.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:21.629 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:02:21.622+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:21.626557+0000) 2019-10-18T00:02:22.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:22.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:22.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:22.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:23.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:23.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:23.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:23.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:24.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:24.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:24.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:24.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:24.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:24.513+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:25.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:25.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:25.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:25.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:26.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:26.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:26.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:26.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:27.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:02:27.057+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:27.058235+0000) 2019-10-18T00:02:27.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:27.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:27.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:27.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:28.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:28.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:28.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:28.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:29.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:02:29.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:29.103176+0000) 2019-10-18T00:02:29.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:29.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:29.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:29.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:29.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:29.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:30.125 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:02:30.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:30.122556+0000) 2019-10-18T00:02:30.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:30.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:30.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:30.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:31.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:31.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:31.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:31.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:31.630 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:02:31.622+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:31.626715+0000) 2019-10-18T00:02:32.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:32.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:32.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:32.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:33.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:33.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:33.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:33.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:34.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:34.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:34.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:34.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:34.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:34.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:35.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:35.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:35.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:35.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:36.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:36.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:36.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:36.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:37.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:02:37.057+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:37.058332+0000) 2019-10-18T00:02:37.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:37.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:37.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:37.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:38.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:38.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:38.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:38.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:39.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:02:39.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:39.103278+0000) 2019-10-18T00:02:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:39.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:39.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:39.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:39.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:40.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:02:40.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:40.122653+0000) 2019-10-18T00:02:40.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:40.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:40.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:40.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:41.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:41.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:41.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:41.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:41.630 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:02:41.622+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:41.626893+0000) 2019-10-18T00:02:42.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:42.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:42.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:42.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:43.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:43.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:43.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:43.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:44.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:44.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:44.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:44.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:44.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:44.516+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:45.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:45.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:45.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:45.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:46.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:46.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:46.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:46.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:47.061 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:02:47.056+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:47.058434+0000) 2019-10-18T00:02:47.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:47.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:47.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:47.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:48.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:48.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:48.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:48.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:49.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:02:49.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:49.103371+0000) 2019-10-18T00:02:49.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:49.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:49.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:49.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:49.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:49.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:50.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:50.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:50.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:50.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:50.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:02:50.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:50.122776+0000) 2019-10-18T00:02:51.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:51.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:51.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:51.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:51.630 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:02:51.622+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:51.627095+0000) 2019-10-18T00:02:52.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:52.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:52.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:52.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:53.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:53.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:53.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:53.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:54.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:54.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:54.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:54.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:54.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:54.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:02:55.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:55.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:55.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:55.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:56.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:56.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:57.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:02:57.056+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:57.058532+0000) 2019-10-18T00:02:57.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:57.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:57.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:57.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:58.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:58.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:59.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:59.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:02:59.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:02:59.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:02:59.106 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:02:59.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:02:59.103473+0000) 2019-10-18T00:02:59.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:02:59.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:00.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:00.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:00.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:00.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:00.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:03:00.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:00.122904+0000) 2019-10-18T00:03:01.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:01.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:01.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:01.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:01.630 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:03:01.626+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:01.627289+0000) 2019-10-18T00:03:02.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:02.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:02.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:02.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:03.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:03.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:03.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:03.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:04.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:04.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:04.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:04.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:04.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:04.520+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:05.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:05.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:05.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:05.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:06.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:06.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:06.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:06.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:07.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:03:07.056+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:07.058635+0000) 2019-10-18T00:03:07.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:07.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:07.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:07.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:08.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:08.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:08.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:08.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:09.107 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:09.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:09.103573+0000) 2019-10-18T00:03:09.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:09.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:09.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:09.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:09.529 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:09.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:10.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:10.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:10.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:10.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:10.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:03:10.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:10.123017+0000) 2019-10-18T00:03:11.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:11.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:11.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:11.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:11.630 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:03:11.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:11.627469+0000) 2019-10-18T00:03:12.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:12.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:12.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:12.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:13.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:13.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:13.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:13.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:14.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:14.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:14.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:14.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:14.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:14.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:15.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:15.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:15.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:15.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:16.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:16.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:16.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:16.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:17.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:03:17.055+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:17.058740+0000) 2019-10-18T00:03:17.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:17.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:17.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:17.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:18.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:18.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:18.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:18.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:19.107 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:19.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:19.103677+0000) 2019-10-18T00:03:19.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:19.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:19.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:19.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:19.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:19.523+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:20.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:03:20.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:20.123136+0000) 2019-10-18T00:03:20.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:20.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:20.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:20.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:21.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:21.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:21.630 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:03:21.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:21.627649+0000) 2019-10-18T00:03:22.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:22.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:22.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:22.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:23.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:23.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:23.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:24.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:24.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:24.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:24.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:24.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:24.523+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:25.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:25.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:25.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:25.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:26.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:26.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:26.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:26.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:27.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:27.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:27.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:27.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:27.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:03:27.055+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:27.058841+0000) 2019-10-18T00:03:28.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:28.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:28.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:28.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:29.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:29.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:29.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:29.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:29.107 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:29.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:29.103780+0000) 2019-10-18T00:03:29.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:29.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:30.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:30.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:30.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:30.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:30.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:03:30.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:30.123251+0000) 2019-10-18T00:03:31.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:31.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:31.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:31.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:31.631 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:03:31.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:31.627833+0000) 2019-10-18T00:03:32.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:32.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:32.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:32.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:33.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:33.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:33.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:33.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:34.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:34.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:34.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:34.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:34.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:34.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:35.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:35.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:35.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:35.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:36.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:36.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:36.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:36.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:37.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:03:37.055+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:37.058951+0000) 2019-10-18T00:03:37.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:37.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:37.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:37.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:38.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:38.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:38.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:38.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:39.107 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:39.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:39.103884+0000) 2019-10-18T00:03:39.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:39.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:39.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:39.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:39.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:39.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:40.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:03:40.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:40.123372+0000) 2019-10-18T00:03:40.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:40.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:40.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:40.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:41.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:41.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:41.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:41.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:41.631 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:03:41.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:41.628010+0000) 2019-10-18T00:03:42.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:42.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:42.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:42.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:43.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:43.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:43.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:43.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:44.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:44.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:44.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:44.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:44.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:44.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:45.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:45.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:45.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:45.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:46.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:46.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:46.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:46.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:47.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:03:47.055+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:47.059054+0000) 2019-10-18T00:03:47.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:47.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:47.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:47.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:48.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:48.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:48.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:48.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:49.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:49.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:49.103982+0000) 2019-10-18T00:03:49.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:49.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:49.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:49.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:49.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:49.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:50.126 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:03:50.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:50.123494+0000) 2019-10-18T00:03:50.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:50.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:50.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:50.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:51.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:51.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:51.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:51.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:51.631 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:03:51.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:51.628190+0000) 2019-10-18T00:03:52.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:52.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:52.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:52.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:53.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:53.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:53.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:53.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:54.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:54.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:54.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:54.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:54.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:54.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:03:55.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:55.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:55.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:55.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:56.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:56.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:56.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:56.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:57.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:03:57.054+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:57.059155+0000) 2019-10-18T00:03:57.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:57.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:57.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:57.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:58.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:58.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:58.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:58.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:59.107 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:59.098+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:59.104081+0000) 2019-10-18T00:03:59.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:03:59.098+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:03:59.105199+0000) 2019-10-18T00:03:59.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:59.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:03:59.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:03:59.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:03:59.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:03:59.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:00.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:04:00.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:00.123619+0000) 2019-10-18T00:04:00.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:00.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:00.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:00.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:01.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:01.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:01.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:01.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:01.632 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:01.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:01.628371+0000) 2019-10-18T00:04:02.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:02.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:02.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:02.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:03.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:03.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:03.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:03.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:03.607 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:04:03.598+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:03.604277+0000) 2019-10-18T00:04:04.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:04.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:04.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:04.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:04.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:04.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:05.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:05.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:05.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:05.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:06.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:06.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:06.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:06.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:07.062 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:07.054+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:07.059252+0000) 2019-10-18T00:04:07.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:07.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:07.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:07.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:08.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:08.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:08.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:08.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:09.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:09.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:09.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:09.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:09.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:09.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:10.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:04:10.121+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:10.123739+0000) 2019-10-18T00:04:10.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:10.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:10.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:10.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:11.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:11.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:11.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:11.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:11.631 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:11.625+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:11.628555+0000) 2019-10-18T00:04:12.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:12.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:12.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:12.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:13.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:13.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:13.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:13.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:13.607 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:04:13.598+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:13.604374+0000) 2019-10-18T00:04:14.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:14.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:14.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:14.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:14.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:14.530+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:15.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:15.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:15.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:15.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:16.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:16.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:16.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:16.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:17.063 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:17.054+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:17.059351+0000) 2019-10-18T00:04:17.063 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:17.054+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:17.060393+0000) 2019-10-18T00:04:17.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:17.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:17.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:17.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:18.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:18.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:18.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:18.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:19.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:19.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:19.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:19.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:19.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:19.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:20.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:04:20.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:20.123857+0000) 2019-10-18T00:04:20.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:20.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:20.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:20.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:21.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:21.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:21.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:21.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:21.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:21.554+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:21.559560+0000) 2019-10-18T00:04:21.632 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:21.624+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:21.628733+0000) 2019-10-18T00:04:22.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:22.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:22.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:22.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:23.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:23.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:23.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:23.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:23.607 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:04:23.598+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:23.604471+0000) 2019-10-18T00:04:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:24.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:24.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:24.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:24.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:25.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:25.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:25.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:25.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:26.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:26.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:26.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:26.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:27.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:27.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:27.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:27.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:28.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:28.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:28.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:28.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:29.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:29.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:29.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:29.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:29.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:29.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:30.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:04:30.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:30.123977+0000) 2019-10-18T00:04:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:30.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:30.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:31.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:31.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:31.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:31.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:31.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:31.553+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:31.559662+0000) 2019-10-18T00:04:31.632 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:31.624+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:31.628916+0000) 2019-10-18T00:04:32.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:32.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:32.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:32.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:33.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:33.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:33.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:33.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:33.608 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:04:33.597+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:33.604569+0000) 2019-10-18T00:04:34.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:34.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:34.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:34.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:34.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:34.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:35.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:35.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:35.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:35.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:36.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:36.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:36.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:36.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:37.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:37.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:37.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:37.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:38.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:38.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:39.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:39.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:39.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:39.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:39.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:39.537+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:40.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:04:40.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:40.124085+0000) 2019-10-18T00:04:40.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:40.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:40.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:40.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:41.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:41.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:41.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:41.553+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:41.559762+0000) 2019-10-18T00:04:41.632 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:41.624+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:41.629124+0000) 2019-10-18T00:04:42.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:42.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:42.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:42.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:43.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:43.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:43.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:43.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:43.608 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:04:43.597+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:43.604664+0000) 2019-10-18T00:04:44.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:44.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:44.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:44.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:44.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:44.537+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:45.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:45.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:45.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:45.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:46.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:46.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:46.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:46.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:47.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:47.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:47.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:47.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:48.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:48.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:48.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:48.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:49.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:49.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:49.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:49.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:49.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:49.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:50.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:04:50.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:50.124206+0000) 2019-10-18T00:04:50.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:50.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:50.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:50.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:51.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:51.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:51.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:51.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:51.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:04:51.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:51.559866+0000) 2019-10-18T00:04:51.632 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:51.624+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:51.629254+0000) 2019-10-18T00:04:51.633 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:51.628+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:51.630237+0000) 2019-10-18T00:04:52.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:52.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:52.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:52.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:53.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:53.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:53.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:53.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:53.608 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:04:53.601+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:53.604761+0000) 2019-10-18T00:04:54.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:54.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:04:54.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:54.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:54.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:54.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:55.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:55.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:55.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:55.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:56.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:04:56.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:04:56.129668+0000) 2019-10-18T00:04:56.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:56.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:56.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:56.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:57.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:57.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:57.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:57.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:58.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:58.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:58.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:58.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:59.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:59.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:04:59.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:04:59.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:04:59.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:04:59.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:00.127 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:05:00.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:00.124334+0000) 2019-10-18T00:05:00.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:00.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:00.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:00.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:01.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:01.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:01.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:01.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:01.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:05:01.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:01.559966+0000) 2019-10-18T00:05:02.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:02.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:02.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:02.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:03.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:03.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:03.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:03.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:03.608 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:03.600+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:03.604861+0000) 2019-10-18T00:05:04.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:04.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:04.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:04.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:04.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:04.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:05.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:05.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:05.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:05.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:06.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:05:06.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:06.129848+0000) 2019-10-18T00:05:06.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:06.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:06.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:06.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:07.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:07.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:07.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:07.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:08.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:08.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:08.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:08.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:09.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:09.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:09.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:09.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:09.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:09.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:10.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:05:10.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:10.124451+0000) 2019-10-18T00:05:10.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:10.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:10.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:10.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:11.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:11.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:11.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:11.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:11.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:05:11.556+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:11.560070+0000) 2019-10-18T00:05:12.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:12.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:12.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:12.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:13.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:13.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:13.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:13.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:13.608 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:13.600+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:13.604956+0000) 2019-10-18T00:05:13.609 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:13.604+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:13.605917+0000) 2019-10-18T00:05:14.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:14.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:14.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:14.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:14.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:14.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:15.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:15.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:15.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:15.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:16.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:05:16.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:16.130034+0000) 2019-10-18T00:05:16.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:16.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:16.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:16.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:17.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:17.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:17.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:17.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:18.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:18.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:18.105153+0000) 2019-10-18T00:05:18.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:18.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:18.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:18.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:19.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:19.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:19.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:19.544+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:20.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:05:20.120+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:20.124554+0000) 2019-10-18T00:05:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:20.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:20.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:21.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:21.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:21.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:21.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:21.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:05:21.556+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:21.560175+0000) 2019-10-18T00:05:22.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:22.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:22.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:22.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:23.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:23.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:23.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:23.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:24.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:24.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:24.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:24.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:24.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:24.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:25.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:25.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:25.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:25.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:26.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:05:26.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:26.130231+0000) 2019-10-18T00:05:26.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:26.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:26.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:26.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:27.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:27.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:27.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:27.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:28.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:28.100+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:28.105257+0000) 2019-10-18T00:05:28.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:28.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:28.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:28.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:29.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:29.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:29.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:29.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:29.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:29.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:30.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:05:30.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:30.124670+0000) 2019-10-18T00:05:30.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:30.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:30.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:30.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:31.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:31.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:31.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:31.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:31.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:05:31.556+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:31.560275+0000) 2019-10-18T00:05:32.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:32.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:32.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:32.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:33.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:33.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:33.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:33.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:34.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:34.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:34.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:34.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:34.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:34.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:35.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:35.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:35.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:35.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:36.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:05:36.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:36.130389+0000) 2019-10-18T00:05:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:36.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:36.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:37.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:37.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:37.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:37.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:38.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:38.099+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:38.105368+0000) 2019-10-18T00:05:38.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:38.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:38.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:38.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:39.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:39.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:39.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:39.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:39.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:39.547+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:40.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:05:40.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:40.124783+0000) 2019-10-18T00:05:40.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:40.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:40.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:40.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:41.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:41.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:41.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:41.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:41.563 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:05:41.555+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:41.560372+0000) 2019-10-18T00:05:42.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:42.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:42.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:42.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:43.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:43.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:43.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:43.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:44.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:44.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:44.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:44.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:44.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:44.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:45.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:45.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:45.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:45.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:46.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:05:46.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:46.130554+0000) 2019-10-18T00:05:46.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:46.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:46.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:46.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:47.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:47.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:47.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:47.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:48.108 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:48.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:48.105469+0000) 2019-10-18T00:05:48.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:48.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:48.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:48.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:49.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:49.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:49.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:49.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:49.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:49.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:50.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:05:50.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:50.124903+0000) 2019-10-18T00:05:50.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:50.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:50.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:50.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:51.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:05:51.555+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:51.560469+0000) 2019-10-18T00:05:51.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:51.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:51.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:51.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:52.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:52.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:52.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:52.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:53.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:53.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:53.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:53.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:54.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:54.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:54.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:54.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:54.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:54.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:55.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:55.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:55.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:56.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:05:56.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:56.130719+0000) 2019-10-18T00:05:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:56.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:56.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:57.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:57.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:57.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:57.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:58.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:05:58.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:05:58.105573+0000) 2019-10-18T00:05:58.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:58.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:58.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:58.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:05:59.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:05:59.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:05:59.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:59.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:05:59.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:05:59.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:00.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:06:00.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:00.125016+0000) 2019-10-18T00:06:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:00.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:00.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:01.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:06:01.555+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:01.560572+0000) 2019-10-18T00:06:01.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:01.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:01.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:01.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:02.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:02.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:02.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:02.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:03.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:03.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:03.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:03.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:04.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:04.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:04.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:04.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:04.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:04.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:05.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:05.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:05.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:05.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:06.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:06:06.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:06.130905+0000) 2019-10-18T00:06:06.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:06.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:06.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:06.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:07.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:07.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:07.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:07.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:08.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:06:08.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:08.105679+0000) 2019-10-18T00:06:08.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:08.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:08.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:08.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:09.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:09.555+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:09.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:09.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:09.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:09.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:10.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:06:10.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:10.125129+0000) 2019-10-18T00:06:10.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:10.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:10.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:10.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:11.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:06:11.555+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:11.560672+0000) 2019-10-18T00:06:11.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:11.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:11.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:11.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:12.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:12.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:12.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:12.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:13.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:13.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:13.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:13.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:14.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:14.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:14.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:14.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:14.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:14.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:15.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:15.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:15.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:15.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:16.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:06:16.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:16.131049+0000) 2019-10-18T00:06:16.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:16.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:16.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:16.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:17.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:17.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:17.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:17.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:18.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:06:18.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:18.105782+0000) 2019-10-18T00:06:18.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:18.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:18.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:18.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:19.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:19.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:19.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:19.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:19.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:19.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:20.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:06:20.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:20.125245+0000) 2019-10-18T00:06:20.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:20.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:20.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:20.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:21.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:06:21.554+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:21.560770+0000) 2019-10-18T00:06:21.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:21.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:21.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:21.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:22.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:22.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:22.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:22.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:23.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:23.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:23.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:23.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:24.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:24.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:24.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:24.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:24.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:24.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:25.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:25.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:25.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:25.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:26.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:06:26.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:26.131225+0000) 2019-10-18T00:06:26.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:26.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:26.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:26.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:27.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:27.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:27.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:27.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:28.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:06:28.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:28.105887+0000) 2019-10-18T00:06:28.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:28.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:28.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:28.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:29.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:29.554+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:29.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:29.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:29.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:29.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:30.133 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:06:30.119+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:30.125364+0000) 2019-10-18T00:06:30.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:30.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:30.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:30.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:31.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:06:31.554+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:31.560874+0000) 2019-10-18T00:06:31.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:31.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:31.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:31.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:32.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:32.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:32.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:32.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:33.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:33.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:33.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:33.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:34.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:34.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:34.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:34.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:34.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:34.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:35.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:35.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:35.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:35.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:36.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:06:36.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:36.131409+0000) 2019-10-18T00:06:36.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:36.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:36.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:36.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:37.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:37.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:37.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:37.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:38.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:06:38.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:38.105991+0000) 2019-10-18T00:06:38.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:38.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:38.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:38.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:39.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:39.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:39.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:39.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:39.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:39.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:40.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:06:40.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:40.125475+0000) 2019-10-18T00:06:40.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:40.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:40.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:40.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:41.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:06:41.554+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:41.560975+0000) 2019-10-18T00:06:41.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:41.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:41.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:41.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:42.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:42.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:42.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:42.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:43.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:43.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:43.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:43.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:44.567 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:44.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:44.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:44.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:44.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:44.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:45.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:45.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:45.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:45.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:46.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:06:46.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:46.131591+0000) 2019-10-18T00:06:46.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:46.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:46.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:46.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:47.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:47.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:47.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:47.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:48.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:06:48.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:48.106090+0000) 2019-10-18T00:06:48.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:48.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:48.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:48.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:49.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:49.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:49.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:49.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:49.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:49.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:50.128 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:06:50.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:50.125593+0000) 2019-10-18T00:06:50.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:50.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:50.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:50.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:51.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:06:51.553+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:51.561077+0000) 2019-10-18T00:06:51.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:51.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:51.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:51.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:52.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:52.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:52.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:52.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:53.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:53.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:53.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:53.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:54.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:54.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:54.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:54.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:54.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:54.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:55.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:55.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:55.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:55.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:56.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:06:56.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:56.131783+0000) 2019-10-18T00:06:56.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:56.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:56.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:56.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:57.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:57.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:57.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:57.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:58.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:06:58.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:06:58.106190+0000) 2019-10-18T00:06:58.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:58.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:58.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:58.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:06:59.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:06:59.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:06:59.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:59.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:06:59.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:06:59.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:00.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:07:00.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:00.125714+0000) 2019-10-18T00:07:00.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:00.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:00.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:00.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:01.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:07:01.553+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:01.561177+0000) 2019-10-18T00:07:01.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:01.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:01.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:01.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:02.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:02.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:02.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:02.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:03.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:03.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:03.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:03.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:04.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:04.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:04.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:04.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:05.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:05.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:05.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:05.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:06.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:07:06.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:06.131981+0000) 2019-10-18T00:07:06.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:06.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:06.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:06.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:07.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:07.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:07.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:07.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:08.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:07:08.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:08.106290+0000) 2019-10-18T00:07:08.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:08.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:08.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:08.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:09.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:09.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:09.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:09.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:09.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:09.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:10.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:07:10.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:10.125826+0000) 2019-10-18T00:07:10.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:10.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:10.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:10.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:11.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:07:11.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:11.561279+0000) 2019-10-18T00:07:11.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:11.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:11.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:11.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:12.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:12.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:12.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:12.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:13.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:13.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:13.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:13.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:14.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:14.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:14.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:14.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:14.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:14.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:15.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:15.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:15.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:15.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:16.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:07:16.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:16.132186+0000) 2019-10-18T00:07:16.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:16.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:16.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:16.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:17.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:17.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:17.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:17.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:18.109 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:07:18.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:18.106393+0000) 2019-10-18T00:07:18.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:18.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:18.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:18.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:19.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:19.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:19.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:19.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:19.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:19.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:20.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:07:20.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:20.125942+0000) 2019-10-18T00:07:20.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:20.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:20.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:20.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:21.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:07:21.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:21.561381+0000) 2019-10-18T00:07:21.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:21.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:21.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:21.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:22.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:22.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:22.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:22.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:23.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:23.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:24.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:24.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:24.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:24.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:24.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:24.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:25.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:25.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:25.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:25.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:26.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:07:26.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:26.132370+0000) 2019-10-18T00:07:26.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:26.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:26.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:26.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:28.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:28.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:28.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:28.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:28.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:07:28.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:28.106498+0000) 2019-10-18T00:07:28.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:28.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:28.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:28.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:29.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:29.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:29.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:29.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:29.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:29.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:30.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:07:30.118+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:30.126066+0000) 2019-10-18T00:07:30.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:30.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:30.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:30.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:31.564 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:07:31.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:31.561463+0000) 2019-10-18T00:07:31.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:31.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:31.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:31.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:32.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:32.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:32.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:32.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:33.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:33.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:33.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:33.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:34.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:34.568+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:34.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:34.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:34.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:34.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:35.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:35.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:35.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:35.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:36.136 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:07:36.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:36.132574+0000) 2019-10-18T00:07:36.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:36.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:36.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:36.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:37.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:37.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:37.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:37.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:38.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:07:38.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:38.106599+0000) 2019-10-18T00:07:38.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:38.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:38.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:38.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:39.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:39.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:39.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:39.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:39.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:39.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:40.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:07:40.126+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:40.126213+0000) 2019-10-18T00:07:40.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:40.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:40.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:40.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:41.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:07:41.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:41.561555+0000) 2019-10-18T00:07:41.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:41.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:41.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:41.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:42.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:42.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:42.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:42.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:43.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:43.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:43.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:43.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:44.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:44.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:44.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:44.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:44.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:44.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:45.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:45.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:45.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:45.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:46.136 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:07:46.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:46.132792+0000) 2019-10-18T00:07:46.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:46.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:46.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:46.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:47.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:47.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:47.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:47.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:48.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:07:48.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:48.106699+0000) 2019-10-18T00:07:48.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:48.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:48.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:48.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:49.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:49.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:49.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:49.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:49.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:49.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:50.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:07:50.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:50.126318+0000) 2019-10-18T00:07:50.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:50.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:50.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:50.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:51.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:07:51.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:51.561656+0000) 2019-10-18T00:07:52.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:51.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:52.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:51.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:53.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:53.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:53.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:53.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:54.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:54.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:54.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:54.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:54.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:54.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:07:55.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:55.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:55.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:55.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:56.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:56.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:56.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:56.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:56.136 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:07:56.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:56.133009+0000) 2019-10-18T00:07:57.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:57.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:57.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:57.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:58.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:58.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:58.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:58.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:58.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:07:58.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:07:58.106804+0000) 2019-10-18T00:07:59.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:59.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:07:59.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:07:59.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:07:59.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:07:59.571+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:00.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:00.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:00.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:00.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:00.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:08:00.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:00.126420+0000) 2019-10-18T00:08:01.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:01.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:01.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:01.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:01.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:08:01.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:01.561757+0000) 2019-10-18T00:08:02.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:02.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:02.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:02.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:03.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:03.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:03.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:03.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:04.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:04.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:04.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:04.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:04.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:04.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:05.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:05.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:05.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:05.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:06.136 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:08:06.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:06.133193+0000) 2019-10-18T00:08:06.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:06.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:06.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:06.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:07.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:07.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:07.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:07.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:08.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:08.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:08.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:08.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:08.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:08:08.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:08.106905+0000) 2019-10-18T00:08:09.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:09.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:09.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:09.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:09.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:09.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:10.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:10.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:10.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:10.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:10.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:08:10.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:10.126518+0000) 2019-10-18T00:08:11.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:11.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:11.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:11.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:11.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:08:11.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:11.561860+0000) 2019-10-18T00:08:12.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:12.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:12.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:12.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:13.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:13.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:13.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:13.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:14.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:14.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:14.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:14.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:14.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:14.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:15.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:15.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:15.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:15.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:16.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:16.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:16.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:16.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:16.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:08:16.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:16.133388+0000) 2019-10-18T00:08:17.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:17.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:17.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:17.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:18.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:08:18.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:18.107007+0000) 2019-10-18T00:08:18.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:18.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:18.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:18.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:19.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:19.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:19.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:19.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:19.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:19.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:20.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:08:20.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:20.126620+0000) 2019-10-18T00:08:20.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:20.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:20.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:20.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:21.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:21.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:21.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:21.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:21.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:08:21.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:21.561956+0000) 2019-10-18T00:08:22.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:22.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:22.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:22.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:23.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:23.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:23.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:23.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:24.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:24.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:24.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:24.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:24.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:24.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:25.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:25.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:25.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:25.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:26.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:08:26.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:26.133570+0000) 2019-10-18T00:08:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:26.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:26.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:27.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:27.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:27.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:27.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:28.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:08:28.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:28.107113+0000) 2019-10-18T00:08:28.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:28.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:28.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:28.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:29.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:29.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:29.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:29.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:29.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:29.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:30.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:30.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:30.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:30.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:30.129 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:08:30.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:30.126722+0000) 2019-10-18T00:08:31.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:31.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:31.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:31.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:31.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:08:31.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:31.562057+0000) 2019-10-18T00:08:32.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:32.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:32.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:32.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:33.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:33.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:33.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:33.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:34.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:34.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:34.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:34.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:34.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:35.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:35.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:35.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:35.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:36.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:36.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:36.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:36.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:36.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:08:36.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:36.133750+0000) 2019-10-18T00:08:37.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:37.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:37.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:37.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:38.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:38.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:38.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:38.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:38.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:08:38.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:38.107254+0000) 2019-10-18T00:08:39.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:39.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:39.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:39.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:39.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:39.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:40.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:40.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:40.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:40.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:40.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:08:40.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:40.126835+0000) 2019-10-18T00:08:41.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:41.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:41.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:41.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:41.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:08:41.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:41.562163+0000) 2019-10-18T00:08:42.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:42.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:42.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:42.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:43.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:43.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:43.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:43.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:44.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:44.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:44.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:44.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:44.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:44.578+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:44.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:44.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:44.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:44.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:46.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:46.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:46.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:46.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:46.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:08:46.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:46.133955+0000) 2019-10-18T00:08:47.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:47.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:47.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:47.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:48.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:48.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:48.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:48.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:48.110 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:08:48.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:48.107352+0000) 2019-10-18T00:08:49.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:49.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:49.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:49.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:49.593 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:49.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:50.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:50.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:50.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:50.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:50.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:08:50.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:50.126950+0000) 2019-10-18T00:08:51.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:51.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:51.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:51.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:51.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:08:51.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:51.562263+0000) 2019-10-18T00:08:52.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:52.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:52.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:52.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:53.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:53.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:53.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:53.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:54.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:54.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:54.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:54.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:54.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:54.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:08:55.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:55.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:55.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:55.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:56.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:08:56.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:56.134156+0000) 2019-10-18T00:08:56.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:56.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:56.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:56.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:57.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:57.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:57.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:57.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:58.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:08:58.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:08:58.107456+0000) 2019-10-18T00:08:58.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:58.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:58.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:58.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:59.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:59.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:08:59.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:08:59.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:08:59.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:08:59.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:00.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:09:00.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:00.127069+0000) 2019-10-18T00:09:00.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:00.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:00.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:00.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:01.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:01.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:01.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:01.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:01.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:09:01.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:01.562361+0000) 2019-10-18T00:09:02.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:02.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:02.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:02.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:03.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:03.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:03.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:03.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:04.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:04.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:04.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:04.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:04.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:04.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:05.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:05.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:05.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:05.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:06.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:09:06.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:06.134331+0000) 2019-10-18T00:09:06.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:06.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:06.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:06.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:07.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:07.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:07.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:07.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:08.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:09:08.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:08.107561+0000) 2019-10-18T00:09:08.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:08.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:08.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:08.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:09.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:09.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:09.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:09.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:09.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:09.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:10.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:09:10.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:10.127182+0000) 2019-10-18T00:09:10.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:10.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:10.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:10.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:11.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:11.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:11.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:11.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:11.565 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:09:11.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:11.562458+0000) 2019-10-18T00:09:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:12.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:12.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:13.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:13.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:13.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:13.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:14.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:14.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:14.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:14.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:14.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:14.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:15.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:15.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:15.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:15.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:16.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:09:16.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:16.134480+0000) 2019-10-18T00:09:16.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:16.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:16.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:16.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:17.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:17.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:17.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:17.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:18.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:09:18.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:18.107660+0000) 2019-10-18T00:09:18.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:18.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:18.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:18.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:19.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:19.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:19.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:19.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:19.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:19.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:20.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:09:20.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:20.127299+0000) 2019-10-18T00:09:20.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:20.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:20.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:20.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:21.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:21.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:21.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:21.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:21.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:09:21.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:21.562558+0000) 2019-10-18T00:09:22.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:22.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:22.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:22.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:23.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:23.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:23.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:23.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:24.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:24.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:24.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:24.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:24.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:24.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:25.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:25.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:25.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:25.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:26.138 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:09:26.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:26.134657+0000) 2019-10-18T00:09:26.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:26.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:26.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:26.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:27.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:27.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:27.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:27.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:28.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:09:28.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:28.107760+0000) 2019-10-18T00:09:28.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:28.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:28.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:28.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:29.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:29.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:29.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:29.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:29.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:29.589+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:30.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:09:30.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:30.127412+0000) 2019-10-18T00:09:30.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:30.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:30.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:30.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:31.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:31.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:31.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:31.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:31.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:09:31.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:31.562655+0000) 2019-10-18T00:09:32.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:32.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:32.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:32.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:33.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:33.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:33.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:33.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:34.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:34.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:34.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:34.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:34.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:34.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:35.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:35.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:35.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:35.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:36.138 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:09:36.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:36.134826+0000) 2019-10-18T00:09:36.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:36.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:36.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:36.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:37.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:37.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:37.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:37.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:38.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:09:38.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:38.107863+0000) 2019-10-18T00:09:38.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:38.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:38.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:38.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:39.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:39.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:39.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:39.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:39.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:39.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:40.130 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:09:40.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:40.127506+0000) 2019-10-18T00:09:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:40.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:40.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:41.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:41.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:41.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:41.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:41.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:09:41.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:41.562752+0000) 2019-10-18T00:09:42.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:42.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:42.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:42.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:43.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:43.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:43.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:43.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:44.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:44.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:44.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:44.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:44.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:44.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:45.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:45.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:45.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:45.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:46.138 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:09:46.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:46.135009+0000) 2019-10-18T00:09:46.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:46.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:46.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:46.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:47.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:47.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:47.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:47.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:48.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:09:48.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:48.107962+0000) 2019-10-18T00:09:48.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:48.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:48.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:48.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:49.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:49.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:49.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:49.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:49.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:49.592+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:50.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:09:50.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:50.127619+0000) 2019-10-18T00:09:50.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:50.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:50.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:50.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:51.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:51.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:51.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:51.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:51.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:09:51.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:51.562852+0000) 2019-10-18T00:09:52.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:52.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:52.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:52.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:53.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:53.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:53.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:53.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:54.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:54.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:54.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:54.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:54.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:54.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:09:55.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:55.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:55.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:55.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:56.138 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:09:56.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:56.135188+0000) 2019-10-18T00:09:56.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:56.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:56.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:56.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:57.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:57.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:57.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:57.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:58.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:09:58.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:09:58.108064+0000) 2019-10-18T00:09:58.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:58.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:58.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:58.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:59.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:59.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:09:59.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:09:59.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:09:59.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:09:59.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:00.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:10:00.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:00.127738+0000) 2019-10-18T00:10:00.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:00.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:00.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:00.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:01.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:01.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:01.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:01.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:01.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:10:01.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:01.562951+0000) 2019-10-18T00:10:02.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:02.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:02.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:02.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:03.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:03.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:03.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:03.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:04.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:04.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:04.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:04.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:04.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:04.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:05.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:05.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:05.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:05.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:06.138 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:10:06.132+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:06.135396+0000) 2019-10-18T00:10:06.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:06.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:06.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:06.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:07.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:07.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:07.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:07.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:08.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:10:08.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:08.108170+0000) 2019-10-18T00:10:08.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:08.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:08.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:08.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:09.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:09.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:09.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:09.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:09.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:09.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:10.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:10:10.124+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:10.127847+0000) 2019-10-18T00:10:10.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:10.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:10.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:10.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:11.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:11.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:11.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:11.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:11.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:10:11.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:11.563049+0000) 2019-10-18T00:10:12.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:12.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:12.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:12.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:13.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:13.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:13.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:13.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:14.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:14.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:14.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:14.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:14.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:14.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:15.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:15.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:15.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:15.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:16.139 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:10:16.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:16.135575+0000) 2019-10-18T00:10:16.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:16.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:16.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:16.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:17.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:17.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:18.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:10:18.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:18.108277+0000) 2019-10-18T00:10:18.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:18.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:18.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:18.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:19.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:19.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:19.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:19.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:19.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:19.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:20.140 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:10:20.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:20.127959+0000) 2019-10-18T00:10:20.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:20.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:20.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:20.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:21.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:21.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:21.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:21.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:21.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:10:21.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:21.563147+0000) 2019-10-18T00:10:22.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:22.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:22.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:22.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:23.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:23.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:23.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:23.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:24.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:24.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:24.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:24.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:24.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:24.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:25.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:25.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:25.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:25.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:26.139 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:10:26.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:26.135786+0000) 2019-10-18T00:10:26.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:26.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:26.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:26.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:27.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:27.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:27.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:27.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:28.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:10:28.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:28.108378+0000) 2019-10-18T00:10:28.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:28.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:28.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:28.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:29.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:29.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:29.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:29.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:29.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:29.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:30.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:10:30.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:30.128074+0000) 2019-10-18T00:10:30.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:30.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:30.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:30.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:31.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:31.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:31.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:31.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:31.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:10:31.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:31.563246+0000) 2019-10-18T00:10:32.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:32.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:32.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:32.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:33.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:33.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:33.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:33.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:34.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:34.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:34.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:34.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:34.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:34.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:35.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:35.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:35.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:35.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:36.139 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:10:36.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:36.135971+0000) 2019-10-18T00:10:36.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:36.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:36.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:36.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:37.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:37.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:37.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:37.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:38.111 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:10:38.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:38.108482+0000) 2019-10-18T00:10:38.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:38.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:38.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:38.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:39.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:39.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:39.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:39.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:39.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:39.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:40.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:10:40.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:40.128185+0000) 2019-10-18T00:10:40.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:40.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:40.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:40.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:41.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:41.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:41.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:41.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:41.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:10:41.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:41.563347+0000) 2019-10-18T00:10:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:42.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:42.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:43.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:43.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:43.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:43.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:44.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:44.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:44.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:44.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:44.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:44.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:45.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:45.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:45.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:45.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:46.139 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:10:46.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:46.136171+0000) 2019-10-18T00:10:46.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:46.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:46.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:46.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:47.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:47.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:47.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:47.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:48.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:10:48.103+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:48.108590+0000) 2019-10-18T00:10:48.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:48.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:48.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:48.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:49.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:49.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:49.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:49.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:49.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:49.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:50.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:10:50.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:50.128299+0000) 2019-10-18T00:10:50.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:50.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:50.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:50.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:51.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:51.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:51.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:51.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:51.566 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:10:51.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:51.563445+0000) 2019-10-18T00:10:52.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:52.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:52.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:52.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:53.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:53.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:53.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:53.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:54.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:54.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:54.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:54.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:54.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:54.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:10:55.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:55.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:55.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:55.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:56.139 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:10:56.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:56.136349+0000) 2019-10-18T00:10:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:56.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:56.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:57.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:57.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:57.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:57.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:58.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:10:58.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:10:58.108692+0000) 2019-10-18T00:10:58.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:58.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:58.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:58.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:59.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:59.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:10:59.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:10:59.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:10:59.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:10:59.602+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:00.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:11:00.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:00.128423+0000) 2019-10-18T00:11:00.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:00.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:00.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:00.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:01.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:01.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:01.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:01.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:01.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:11:01.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:01.563542+0000) 2019-10-18T00:11:02.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:02.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:02.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:02.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:03.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:03.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:03.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:03.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:04.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:04.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:04.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:04.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:04.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:04.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:05.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:05.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:05.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:05.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:06.139 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:11:06.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:06.136520+0000) 2019-10-18T00:11:06.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:06.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:06.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:06.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:07.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:07.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:07.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:07.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:08.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:11:08.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:08.108800+0000) 2019-10-18T00:11:08.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:08.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:08.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:08.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:09.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:09.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:09.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:09.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:09.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:09.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:10.131 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:11:10.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:10.128550+0000) 2019-10-18T00:11:10.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:10.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:10.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:10.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:11.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:11.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:11.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:11.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:11.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:11:11.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:11.563644+0000) 2019-10-18T00:11:12.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:12.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:12.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:12.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:13.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:13.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:13.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:13.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:14.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:14.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:00:41.573017+0000 (oldest deadline 2019-10-18T00:01:01.573017+0000) 2019-10-18T00:11:14.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:14.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:14.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:14.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:15.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:15.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:16.140 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:11:16.131+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:16.136731+0000) 2019-10-18T00:11:16.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:16.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:17.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:17.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:18.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:11:18.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:18.108905+0000) 2019-10-18T00:11:18.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:18.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:19.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:19.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:19.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:19.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:20.132 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:11:20.123+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:20.128666+0000) 2019-10-18T00:11:20.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:20.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:21.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:21.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:21.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:11:21.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:21.563744+0000) 2019-10-18T00:11:22.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:22.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:23.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:23.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:24.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:24.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:24.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:24.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:25.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:25.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:26.140 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:11:26.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:26.136926+0000) 2019-10-18T00:11:26.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:26.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:27.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:27.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:28.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:11:28.102+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:28.109003+0000) 2019-10-18T00:11:28.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:28.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:29.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:29.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:29.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:29.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:30.132 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:11:30.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:30.128779+0000) 2019-10-18T00:11:30.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:30.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:31.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:31.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:31.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:11:31.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:31.563844+0000) 2019-10-18T00:11:32.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:32.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:33.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:33.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:34.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:34.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:34.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:34.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:35.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:35.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:36.140 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:11:36.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:36.137107+0000) 2019-10-18T00:11:36.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:36.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:36.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:36.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:37.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:37.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:37.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:38.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:11:38.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:38.109107+0000) 2019-10-18T00:11:38.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:38.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:38.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:38.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:39.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:39.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:39.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:39.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:39.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:39.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:40.133 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:11:40.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:40.128899+0000) 2019-10-18T00:11:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:40.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:40.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:41.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:41.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:41.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:41.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:41.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:11:41.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:41.563942+0000) 2019-10-18T00:11:42.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:42.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:42.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:42.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:43.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:43.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:43.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:43.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:44.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:44.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:44.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:44.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:44.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:44.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:45.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:45.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:45.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:45.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:46.140 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:11:46.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:46.137310+0000) 2019-10-18T00:11:46.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:46.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:46.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:46.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:47.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:47.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:47.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:47.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:48.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:11:48.101+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:48.109212+0000) 2019-10-18T00:11:48.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:48.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:48.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:48.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:49.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:49.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:49.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:49.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:49.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:49.613+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:50.132 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:11:50.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:50.129056+0000) 2019-10-18T00:11:50.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:50.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:50.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:50.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:51.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:51.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:51.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:51.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:51.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:11:51.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:51.564038+0000) 2019-10-18T00:11:52.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:52.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:52.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:52.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:53.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:53.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:53.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:53.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:54.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:54.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:54.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:54.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:54.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:54.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:11:55.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:55.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:55.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:55.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:56.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:11:56.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:56.137498+0000) 2019-10-18T00:11:56.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:56.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:56.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:56.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:57.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:57.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:57.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:57.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:58.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:11:58.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:11:58.109312+0000) 2019-10-18T00:11:58.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:58.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:58.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:58.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:59.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:59.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:11:59.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:11:59.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:11:59.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:11:59.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:00.132 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:12:00.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:00.129238+0000) 2019-10-18T00:12:00.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:00.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:00.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:00.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:01.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:01.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:01.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:01.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:01.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:12:01.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:01.564139+0000) 2019-10-18T00:12:02.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:02.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:02.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:02.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:03.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:03.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:03.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:03.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:04.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:04.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:04.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:04.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:04.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:04.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:05.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:05.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:05.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:05.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:06.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:12:06.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:06.137678+0000) 2019-10-18T00:12:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:06.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:06.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:07.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:07.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:07.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:07.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:08.112 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:12:08.104+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:08.109420+0000) 2019-10-18T00:12:08.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:08.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:08.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:08.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:09.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:09.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:09.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:09.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:09.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:09.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:10.132 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:12:10.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:10.129405+0000) 2019-10-18T00:12:10.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:10.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:10.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:10.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:11.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:11.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:11.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:11.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:11.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:12:11.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:11.564242+0000) 2019-10-18T00:12:12.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:12.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:12.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:12.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:13.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:13.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:13.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:13.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:14.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:14.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:14.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:14.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:14.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:14.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:15.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:15.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:16.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:12:16.130+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:16.137850+0000) 2019-10-18T00:12:16.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:16.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:16.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:16.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:17.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:17.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:17.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:17.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:18.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:12:18.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:18.109536+0000) 2019-10-18T00:12:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:18.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:18.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:19.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:19.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:19.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:19.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:19.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:19.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:20.132 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:12:20.122+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:20.129584+0000) 2019-10-18T00:12:20.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:20.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:20.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:20.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:21.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:21.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:21.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:21.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:21.567 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:12:21.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:21.564338+0000) 2019-10-18T00:12:22.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:22.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:22.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:22.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:23.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:23.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:23.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:23.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:24.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:24.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:24.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:24.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:24.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:24.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:25.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:25.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:25.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:25.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:26.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:12:26.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:26.138014+0000) 2019-10-18T00:12:26.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:26.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:26.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:26.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:27.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:27.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:28.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:12:28.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:28.109633+0000) 2019-10-18T00:12:28.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:28.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:28.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:28.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:29.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:29.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:29.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:29.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:29.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:29.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:30.133 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:12:30.126+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:30.129752+0000) 2019-10-18T00:12:30.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:30.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:30.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:30.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:31.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:31.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:31.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:31.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:31.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:12:31.560+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:31.564439+0000) 2019-10-18T00:12:32.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:32.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:32.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:32.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:33.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:33.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:33.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:33.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:34.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:34.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:34.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:34.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:34.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:34.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:35.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:35.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:35.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:35.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:36.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:12:36.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:36.138219+0000) 2019-10-18T00:12:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:36.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:36.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:37.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:37.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:37.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:37.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:38.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:12:38.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:38.109737+0000) 2019-10-18T00:12:38.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:38.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:38.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:38.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:39.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:39.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:39.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:39.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:39.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:39.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:40.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:12:40.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:40.129915+0000) 2019-10-18T00:12:40.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:40.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:40.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:40.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:41.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:41.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:41.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:41.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:12:41.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:41.564541+0000) 2019-10-18T00:12:42.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:42.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:42.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:42.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:43.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:43.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:43.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:43.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:44.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:44.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:44.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:44.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:44.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:44.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:45.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:45.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:45.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:45.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:46.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:12:46.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:46.138378+0000) 2019-10-18T00:12:46.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:46.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:46.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:46.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:47.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:47.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:47.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:47.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:48.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:12:48.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:48.109843+0000) 2019-10-18T00:12:48.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:48.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:48.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:48.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:49.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:49.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:49.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:49.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:49.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:49.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:50.133 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:12:50.125+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:50.130090+0000) 2019-10-18T00:12:50.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:50.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:50.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:50.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:51.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:51.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:51.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:51.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:51.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:12:51.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:51.564639+0000) 2019-10-18T00:12:52.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:52.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:52.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:52.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:53.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:53.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:53.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:53.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:54.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:54.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:54.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:54.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:54.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:54.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:12:55.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:55.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:55.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:55.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:56.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:12:56.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:56.138527+0000) 2019-10-18T00:12:56.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:56.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:56.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:56.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:57.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:57.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:57.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:57.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:58.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:12:58.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:12:58.109939+0000) 2019-10-18T00:12:58.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:58.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:58.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:58.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:59.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:59.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:12:59.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:12:59.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:12:59.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:12:59.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:00.133 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:13:00.129+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:00.130265+0000) 2019-10-18T00:13:00.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:00.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:00.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:00.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:01.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:01.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:01.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:01.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:01.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:13:01.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:01.564735+0000) 2019-10-18T00:13:02.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:02.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:02.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:02.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:03.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:03.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:03.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:03.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:04.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:04.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:04.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:04.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:04.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:04.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:05.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:05.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:05.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:05.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:06.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:13:06.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:06.138696+0000) 2019-10-18T00:13:06.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:06.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:06.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:06.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:07.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:07.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:07.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:07.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:08.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:13:08.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:08.110037+0000) 2019-10-18T00:13:08.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:08.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:08.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:08.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:09.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:09.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:09.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:09.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:09.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:09.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:10.133 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:13:10.129+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:10.130412+0000) 2019-10-18T00:13:10.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:10.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:10.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:10.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:11.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:11.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:11.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:11.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:11.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:13:11.559+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:11.564838+0000) 2019-10-18T00:13:12.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:12.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:12.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:12.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:13.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:13.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:13.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:13.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:14.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:14.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:14.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:14.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:14.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:14.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:15.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:15.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:15.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:15.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:16.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:13:16.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:16.138861+0000) 2019-10-18T00:13:16.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:16.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:16.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:16.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:17.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:17.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:18.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:13:18.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:18.110143+0000) 2019-10-18T00:13:18.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:18.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:18.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:18.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:19.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:19.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:19.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:19.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:19.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:19.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:20.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:13:20.129+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:20.130620+0000) 2019-10-18T00:13:20.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:20.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:20.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:20.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:21.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:21.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:21.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:21.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:21.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:13:21.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:21.564935+0000) 2019-10-18T00:13:22.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:22.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:22.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:22.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:23.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:23.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:23.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:23.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:24.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:24.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:24.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:24.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:24.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:24.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:25.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:25.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:25.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:25.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:26.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:13:26.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:26.139044+0000) 2019-10-18T00:13:26.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:26.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:26.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:26.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:27.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:27.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:27.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:27.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:28.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:13:28.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:28.110246+0000) 2019-10-18T00:13:28.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:28.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:28.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:28.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:29.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:29.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:29.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:29.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:29.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:29.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:30.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:13:30.129+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:30.130797+0000) 2019-10-18T00:13:30.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:30.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:30.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:30.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:31.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:31.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:31.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:31.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:31.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:13:31.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:31.565035+0000) 2019-10-18T00:13:32.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:32.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:32.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:32.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:33.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:33.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:33.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:33.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:34.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:34.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:34.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:34.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:35.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:35.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:35.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:35.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:36.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:13:36.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:36.139226+0000) 2019-10-18T00:13:36.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:36.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:36.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:36.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:37.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:37.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:37.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:37.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:38.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:13:38.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:38.110353+0000) 2019-10-18T00:13:38.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:38.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:38.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:38.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:39.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:39.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:39.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:39.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:39.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:39.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:40.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:13:40.129+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:40.130963+0000) 2019-10-18T00:13:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:40.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:40.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:41.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:41.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:41.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:41.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:41.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:13:41.558+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:41.565133+0000) 2019-10-18T00:13:42.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:42.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:42.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:42.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:43.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:43.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:43.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:43.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:44.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:44.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:44.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:44.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:44.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:44.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:45.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:45.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:45.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:45.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:46.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:13:46.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:46.139410+0000) 2019-10-18T00:13:46.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:46.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:46.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:46.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:47.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:47.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:47.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:47.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:48.113 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:13:48.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:48.110459+0000) 2019-10-18T00:13:48.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:48.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:48.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:48.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:49.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:49.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:49.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:49.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:49.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:49.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:50.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:13:50.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:50.131120+0000) 2019-10-18T00:13:50.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:50.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:50.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:50.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:51.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:51.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:51.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:51.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:51.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:13:51.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:51.565232+0000) 2019-10-18T00:13:52.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:52.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:52.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:52.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:53.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:53.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:53.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:53.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:54.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:54.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:54.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:54.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:54.644 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:54.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:13:55.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:55.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:55.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:55.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:56.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:13:56.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:56.139596+0000) 2019-10-18T00:13:56.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:56.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:56.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:56.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:57.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:57.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:57.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:57.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:58.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:13:58.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:13:58.110560+0000) 2019-10-18T00:13:58.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:58.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:58.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:58.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:59.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:59.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:13:59.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:13:59.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:13:59.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:13:59.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:00.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:14:00.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:00.131292+0000) 2019-10-18T00:14:00.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:00.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:00.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:00.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:01.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:01.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:01.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:01.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:01.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:14:01.557+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:01.565334+0000) 2019-10-18T00:14:02.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:02.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:02.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:02.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:03.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:03.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:03.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:03.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:04.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:04.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:04.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:04.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:04.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:04.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:05.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:05.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:05.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:05.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:06.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:14:06.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:06.139779+0000) 2019-10-18T00:14:06.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:06.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:06.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:06.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:07.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:07.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:08.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:14:08.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:08.110659+0000) 2019-10-18T00:14:08.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:08.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:08.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:08.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:09.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:09.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:09.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:09.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:09.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:09.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:10.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:14:10.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:10.131468+0000) 2019-10-18T00:14:10.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:10.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:10.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:10.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:11.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:11.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:11.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:11.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:11.568 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:14:11.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:11.565434+0000) 2019-10-18T00:14:12.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:12.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:12.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:12.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:13.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:13.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:13.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:13.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:14.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:14.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:14.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:14.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:14.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:14.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:15.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:15.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:15.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:15.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:16.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:14:16.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:16.139959+0000) 2019-10-18T00:14:16.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:16.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:16.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:16.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:17.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:17.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:17.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:17.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:18.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:14:18.109+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:18.110761+0000) 2019-10-18T00:14:18.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:18.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:18.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:18.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:19.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:19.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:19.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:19.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:19.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:19.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:20.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:20.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:20.134 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:14:20.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:20.131646+0000) 2019-10-18T00:14:21.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:21.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:21.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:21.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:21.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:14:21.565+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:21.565536+0000) 2019-10-18T00:14:22.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:22.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:22.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:22.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:23.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:23.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:23.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:23.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:24.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:24.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:24.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:24.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:24.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:24.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:25.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:25.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:25.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:25.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:26.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:14:26.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:26.140139+0000) 2019-10-18T00:14:26.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:26.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:26.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:26.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:27.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:27.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:27.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:27.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:28.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:14:28.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:28.110866+0000) 2019-10-18T00:14:28.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:28.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:28.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:28.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:29.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:29.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:29.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:29.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:29.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:29.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:30.135 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:14:30.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:30.131832+0000) 2019-10-18T00:14:30.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:30.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:30.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:30.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:31.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:31.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:31.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:31.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:31.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:14:31.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:31.565632+0000) 2019-10-18T00:14:32.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:32.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:32.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:32.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:33.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:33.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:33.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:33.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:34.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:34.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:34.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:34.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:34.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:34.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:35.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:35.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:35.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:35.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:36.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:14:36.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:36.140332+0000) 2019-10-18T00:14:36.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:36.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:36.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:36.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:37.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:37.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:37.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:37.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:38.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:14:38.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:38.110967+0000) 2019-10-18T00:14:38.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:38.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:38.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:38.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:39.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:39.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:39.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:39.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:39.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:39.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:40.135 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:14:40.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:40.132012+0000) 2019-10-18T00:14:40.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:40.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:40.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:40.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:41.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:41.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:41.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:41.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:41.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:14:41.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:41.565732+0000) 2019-10-18T00:14:42.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:42.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:42.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:42.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:43.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:43.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:43.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:43.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:44.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:44.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:44.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:44.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:44.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:44.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:45.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:45.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:45.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:45.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:46.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:14:46.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:46.140548+0000) 2019-10-18T00:14:46.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:46.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:46.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:46.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:47.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:47.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:47.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:47.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:48.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:14:48.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:48.111074+0000) 2019-10-18T00:14:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:48.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:48.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:49.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:49.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:49.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:49.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:49.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:49.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:50.135 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:14:50.128+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:50.132180+0000) 2019-10-18T00:14:50.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:50.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:50.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:50.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:51.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:51.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:51.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:51.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:51.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:14:51.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:51.565824+0000) 2019-10-18T00:14:52.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:52.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:52.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:52.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:53.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:53.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:53.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:53.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:54.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:54.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:54.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:54.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:54.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:54.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:14:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:55.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:55.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:56.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:56.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:56.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:56.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:56.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:14:56.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:56.140690+0000) 2019-10-18T00:14:57.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:57.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:57.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:57.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:58.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:14:58.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:14:58.111179+0000) 2019-10-18T00:14:58.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:58.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:58.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:58.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:59.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:59.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:14:59.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:14:59.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:14:59.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:14:59.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:00.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:00.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:00.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:00.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:00.135 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:15:00.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:00.132333+0000) 2019-10-18T00:15:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:01.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:01.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:01.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:15:01.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:01.565920+0000) 2019-10-18T00:15:02.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:02.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:02.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:02.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:03.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:03.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:03.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:03.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:04.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:04.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:04.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:04.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:04.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:04.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:05.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:05.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:05.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:05.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:06.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:06.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:06.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:06.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:06.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:15:06.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:06.140876+0000) 2019-10-18T00:15:07.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:07.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:07.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:07.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:08.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:15:08.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:08.111297+0000) 2019-10-18T00:15:08.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:08.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:08.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:08.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:09.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:09.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:09.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:09.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:09.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:09.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:10.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:10.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:10.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:10.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:10.135 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:15:10.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:10.132486+0000) 2019-10-18T00:15:11.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:11.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:11.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:11.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:11.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:15:11.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:11.566020+0000) 2019-10-18T00:15:12.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:12.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:12.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:12.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:13.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:13.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:13.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:13.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:14.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:14.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:14.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:14.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:14.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:14.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:15.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:15.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:15.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:15.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:16.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:16.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:16.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:16.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:16.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:15:16.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:16.141067+0000) 2019-10-18T00:15:17.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:17.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:17.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:17.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:18.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:18.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:18.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:18.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:18.114 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:15:18.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:18.111408+0000) 2019-10-18T00:15:19.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:19.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:19.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:19.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:19.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:19.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:20.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:20.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:20.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:20.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:20.136 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:15:20.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:20.132677+0000) 2019-10-18T00:15:21.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:21.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:21.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:21.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:21.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:15:21.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:21.566120+0000) 2019-10-18T00:15:22.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:22.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:22.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:22.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:23.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:23.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:23.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:23.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:24.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:24.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:24.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:24.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:24.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:24.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:25.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:25.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:25.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:25.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:26.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:26.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:26.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:26.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:26.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:15:26.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:26.141261+0000) 2019-10-18T00:15:27.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:26.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:27.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:26.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:28.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:27.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:28.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:27.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:28.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:15:28.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:28.111518+0000) 2019-10-18T00:15:29.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:28.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:29.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:28.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:29.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:29.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:30.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:30.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:30.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:30.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:30.136 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:15:30.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:30.132851+0000) 2019-10-18T00:15:31.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:31.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:31.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:31.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:31.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:15:31.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:31.566219+0000) 2019-10-18T00:15:32.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:32.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:32.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:32.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:33.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:33.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:33.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:33.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:34.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:34.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:34.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:34.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:34.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:34.655+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:35.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:35.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:35.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:35.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:36.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:36.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:36.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:36.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:36.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:15:36.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:36.141482+0000) 2019-10-18T00:15:37.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:37.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:37.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:37.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:38.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:15:38.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:38.111626+0000) 2019-10-18T00:15:38.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:38.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:38.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:38.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:39.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:39.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:39.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:39.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:40.136 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:15:40.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:40.133012+0000) 2019-10-18T00:15:40.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:40.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:40.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:40.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:41.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:41.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:41.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:41.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:41.569 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:15:41.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:41.566289+0000) 2019-10-18T00:15:42.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:42.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:42.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:42.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:43.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:43.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:43.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:43.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:44.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:44.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:44.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:44.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:44.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:44.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:45.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:45.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:45.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:45.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:46.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:15:46.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:46.141673+0000) 2019-10-18T00:15:46.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:46.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:46.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:46.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:47.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:47.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:47.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:47.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:48.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:15:48.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:48.111731+0000) 2019-10-18T00:15:48.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:48.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:48.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:48.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:49.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:49.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:49.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:49.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:49.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:49.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:50.136 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:15:50.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:50.133204+0000) 2019-10-18T00:15:50.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:50.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:50.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:50.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:51.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:51.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:51.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:51.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:51.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:15:51.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:51.566387+0000) 2019-10-18T00:15:52.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:52.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:52.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:52.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:53.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:53.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:53.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:53.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:54.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:54.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:54.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:54.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:54.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:54.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:15:55.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:55.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:55.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:55.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:56.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:15:56.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:56.141866+0000) 2019-10-18T00:15:56.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:56.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:56.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:56.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:57.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:57.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:57.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:57.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:58.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:15:58.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:15:58.111835+0000) 2019-10-18T00:15:58.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:58.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:58.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:58.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:59.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:59.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:15:59.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:15:59.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:15:59.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:15:59.658+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:00.136 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:16:00.127+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:00.133390+0000) 2019-10-18T00:16:00.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:00.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:00.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:00.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:01.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:01.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:01.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:01.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:01.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:16:01.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:01.566487+0000) 2019-10-18T00:16:02.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:02.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:02.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:02.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:03.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:03.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:04.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:04.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:04.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:04.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:04.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:04.658+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:05.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:05.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:05.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:05.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:06.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:16:06.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:06.142061+0000) 2019-10-18T00:16:06.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:06.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:06.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:06.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:07.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:07.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:07.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:07.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:08.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:16:08.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:08.111942+0000) 2019-10-18T00:16:08.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:08.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:08.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:08.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:09.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:09.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:09.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:09.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:09.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:09.658+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:10.137 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:16:10.126+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:10.133580+0000) 2019-10-18T00:16:10.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:10.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:10.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:10.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:11.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:11.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:11.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:11.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:11.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:16:11.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:11.566587+0000) 2019-10-18T00:16:12.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:12.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:12.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:12.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:13.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:13.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:13.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:13.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:14.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:14.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:14.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:14.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:14.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:14.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:15.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:15.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:15.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:15.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:16.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:16:16.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:16.142264+0000) 2019-10-18T00:16:16.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:16.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:16.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:16.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:17.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:17.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:17.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:17.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:18.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:16:18.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:18.112054+0000) 2019-10-18T00:16:18.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:18.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:18.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:18.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:19.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:19.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:19.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:19.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:19.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:19.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:20.137 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:16:20.126+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:20.133744+0000) 2019-10-18T00:16:20.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:20.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:20.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:20.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:21.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:21.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:21.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:21.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:21.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:16:21.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:21.566681+0000) 2019-10-18T00:16:22.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:22.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:22.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:22.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:23.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:23.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:23.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:23.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:24.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:24.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:24.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:24.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:24.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:24.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:25.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:25.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:25.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:25.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:26.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:16:26.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:26.142419+0000) 2019-10-18T00:16:26.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:26.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:26.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:26.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:27.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:27.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:27.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:27.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:28.115 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:16:28.105+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:28.112167+0000) 2019-10-18T00:16:28.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:28.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:28.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:28.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:29.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:29.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:29.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:29.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:29.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:29.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:30.137 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:16:30.126+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:30.133919+0000) 2019-10-18T00:16:30.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:30.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:30.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:30.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:31.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:31.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:31.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:31.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:31.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:16:31.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:31.566781+0000) 2019-10-18T00:16:32.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:32.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:32.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:32.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:33.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:33.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:33.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:33.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:34.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:34.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:34.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:34.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:34.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:34.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:35.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:35.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:35.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:35.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:36.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:16:36.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:36.142586+0000) 2019-10-18T00:16:36.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:36.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:36.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:36.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:37.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:37.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:37.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:37.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:38.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:16:38.109+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:38.112275+0000) 2019-10-18T00:16:38.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:38.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:38.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:38.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:39.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:39.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:39.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:39.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:39.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:39.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:40.137 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:16:40.126+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:40.134090+0000) 2019-10-18T00:16:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:40.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:40.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:41.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:41.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:41.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:41.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:41.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:16:41.565+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:41.566880+0000) 2019-10-18T00:16:42.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:42.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:42.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:42.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:43.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:43.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:43.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:43.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:44.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:44.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:44.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:44.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:44.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:44.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:45.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:45.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:45.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:45.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:46.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:16:46.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:46.142785+0000) 2019-10-18T00:16:46.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:46.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:46.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:46.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:47.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:47.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:47.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:47.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:48.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:16:48.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:48.112379+0000) 2019-10-18T00:16:48.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:48.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:48.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:48.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:49.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:49.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:49.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:49.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:49.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:49.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:50.137 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:16:50.130+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:50.134266+0000) 2019-10-18T00:16:50.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:50.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:50.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:50.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:51.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:51.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:51.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:51.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:51.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:16:51.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:51.566983+0000) 2019-10-18T00:16:52.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:52.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:52.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:52.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:53.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:53.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:53.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:53.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:54.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:54.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:54.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:54.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:54.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:54.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:16:55.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:55.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:55.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:55.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:56.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:16:56.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:56.142959+0000) 2019-10-18T00:16:56.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:56.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:56.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:56.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:57.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:57.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:57.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:57.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:58.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:16:58.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:16:58.112494+0000) 2019-10-18T00:16:58.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:58.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:58.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:58.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:59.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:59.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:16:59.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:16:59.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:16:59.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:16:59.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:00.137 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:17:00.130+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:00.134473+0000) 2019-10-18T00:17:00.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:00.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:00.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:00.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:01.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:01.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:01.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:01.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:01.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:17:01.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:01.567084+0000) 2019-10-18T00:17:02.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:02.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:02.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:02.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:03.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:03.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:03.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:03.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:04.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:04.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:04.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:04.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:04.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:04.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:05.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:05.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:05.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:05.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:06.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:17:06.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:06.143140+0000) 2019-10-18T00:17:06.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:06.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:06.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:06.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:07.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:07.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:07.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:07.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:08.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:17:08.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:08.112614+0000) 2019-10-18T00:17:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:08.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:08.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:08.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:09.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:09.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:09.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:09.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:09.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:09.672+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:10.138 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:17:10.134+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:10.134665+0000) 2019-10-18T00:17:10.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:10.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:10.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:10.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:11.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:11.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:11.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:11.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:11.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:17:11.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:11.567184+0000) 2019-10-18T00:17:12.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:12.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:12.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:12.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:13.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:13.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:13.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:13.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:14.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:14.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:14.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:14.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:14.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:14.672+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:15.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:15.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:15.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:15.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:16.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:17:16.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:16.143325+0000) 2019-10-18T00:17:16.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:16.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:16.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:16.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:17.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:17.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:17.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:17.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:18.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:17:18.108+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:18.112723+0000) 2019-10-18T00:17:18.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:18.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:18.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:18.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:19.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:19.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:19.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:19.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:19.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:19.672+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:20.138 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:17:20.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:20.134840+0000) 2019-10-18T00:17:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:20.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:20.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:21.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:21.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:21.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:17:21.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:21.567283+0000) 2019-10-18T00:17:22.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:22.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:22.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:22.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:23.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:23.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:23.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:23.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:24.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:24.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:24.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:24.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:24.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:24.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:25.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:25.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:25.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:25.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:26.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:17:26.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:26.143504+0000) 2019-10-18T00:17:26.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:26.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:26.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:26.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:27.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:27.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:27.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:27.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:28.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:17:28.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:28.112836+0000) 2019-10-18T00:17:28.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:28.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:28.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:28.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:29.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:29.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:29.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:29.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:29.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:29.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:30.138 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:17:30.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:30.135034+0000) 2019-10-18T00:17:30.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:30.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:30.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:30.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:31.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:31.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:31.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:31.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:31.570 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:17:31.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:31.567387+0000) 2019-10-18T00:17:32.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:32.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:32.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:32.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:33.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:33.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:33.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:33.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:34.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:34.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:34.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:34.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:34.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:34.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:35.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:35.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:35.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:35.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:36.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:17:36.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:36.143684+0000) 2019-10-18T00:17:36.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:36.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:36.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:36.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:37.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:37.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:37.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:37.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:38.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:17:38.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:38.112947+0000) 2019-10-18T00:17:38.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:38.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:38.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:38.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:39.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:39.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:39.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:39.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:39.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:39.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:40.138 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:17:40.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:40.135192+0000) 2019-10-18T00:17:40.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:40.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:40.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:40.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:41.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:41.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:41.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:41.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:41.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:17:41.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:41.567484+0000) 2019-10-18T00:17:42.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:42.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:42.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:42.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:43.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:43.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:43.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:43.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:44.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:44.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:44.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:44.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:44.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:44.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:45.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:45.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:45.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:45.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:46.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:17:46.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:46.143896+0000) 2019-10-18T00:17:46.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:46.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:46.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:46.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:47.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:47.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:47.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:47.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:48.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:17:48.107+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:48.113054+0000) 2019-10-18T00:17:48.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:48.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:48.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:48.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:49.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:49.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:49.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:49.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:49.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:49.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:50.138 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:17:50.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:50.135357+0000) 2019-10-18T00:17:50.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:50.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:50.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:50.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:51.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:51.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:51.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:51.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:51.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:17:51.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:51.567582+0000) 2019-10-18T00:17:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:52.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:52.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:53.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:53.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:53.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:53.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:54.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:54.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:54.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:54.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:54.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:54.679+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:17:55.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:55.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:55.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:55.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:56.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:17:56.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:56.144076+0000) 2019-10-18T00:17:56.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:56.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:56.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:56.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:57.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:57.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:57.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:57.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:58.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:17:58.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:17:58.113153+0000) 2019-10-18T00:17:58.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:58.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:58.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:58.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:59.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:59.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:17:59.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:17:59.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:17:59.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:17:59.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:00.138 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:18:00.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:00.135510+0000) 2019-10-18T00:18:00.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:00.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:00.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:00.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:01.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:01.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:01.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:18:01.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:01.567682+0000) 2019-10-18T00:18:02.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:02.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:02.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:02.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:03.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:03.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:03.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:03.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:04.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:04.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:04.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:04.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:04.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:04.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:05.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:05.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:05.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:05.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:06.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:18:06.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:06.144289+0000) 2019-10-18T00:18:06.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:06.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:06.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:06.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:07.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:07.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:07.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:07.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:08.116 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:18:08.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:08.113253+0000) 2019-10-18T00:18:08.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:08.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:08.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:08.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:09.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:09.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:09.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:09.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:09.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:09.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:10.141 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:18:10.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:10.135678+0000) 2019-10-18T00:18:10.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:10.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:10.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:10.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:11.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:11.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:11.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:18:11.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:11.567783+0000) 2019-10-18T00:18:12.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:12.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:12.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:12.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:13.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:13.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:13.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:13.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:14.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:14.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:14.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:14.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:14.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:14.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:15.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:15.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:15.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:15.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:16.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:18:16.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:16.144472+0000) 2019-10-18T00:18:16.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:16.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:16.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:16.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:17.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:17.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:17.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:17.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:18.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:18:18.106+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:18.113364+0000) 2019-10-18T00:18:18.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:18.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:18.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:18.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:19.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:19.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:19.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:19.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:19.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:19.682+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:20.139 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:18:20.133+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:20.135840+0000) 2019-10-18T00:18:20.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:20.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:20.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:20.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:21.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:21.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:21.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:21.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:21.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:18:21.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:21.567879+0000) 2019-10-18T00:18:22.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:22.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:22.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:22.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:23.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:23.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:23.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:23.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:24.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:24.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:24.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:24.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:24.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:24.682+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:25.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:25.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:25.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:25.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:26.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:18:26.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:26.144651+0000) 2019-10-18T00:18:26.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:26.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:26.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:26.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:27.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:27.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:27.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:27.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:28.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:18:28.110+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:28.113473+0000) 2019-10-18T00:18:28.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:28.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:28.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:28.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:29.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:29.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:29.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:29.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:29.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:29.682+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:30.139 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:18:30.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:30.135993+0000) 2019-10-18T00:18:30.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:30.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:30.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:30.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:31.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:18:31.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:31.567980+0000) 2019-10-18T00:18:31.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:31.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:31.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:31.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:32.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:32.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:32.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:32.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:33.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:33.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:33.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:33.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:34.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:34.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:34.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:34.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:34.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:34.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:35.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:35.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:35.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:35.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:36.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:18:36.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:36.144840+0000) 2019-10-18T00:18:36.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:36.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:36.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:36.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:37.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:37.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:37.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:37.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:38.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:18:38.109+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:38.113585+0000) 2019-10-18T00:18:38.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:38.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:38.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:38.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:39.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:39.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:39.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:39.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:39.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:39.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:40.139 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:18:40.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:40.136156+0000) 2019-10-18T00:18:40.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:40.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:40.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:40.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:41.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:41.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:41.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:41.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:41.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:18:41.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:41.568082+0000) 2019-10-18T00:18:42.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:42.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:42.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:42.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:43.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:43.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:43.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:43.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:44.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:44.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:44.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:44.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:44.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:44.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:45.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:45.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:45.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:45.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:46.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:18:46.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:46.145014+0000) 2019-10-18T00:18:46.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:46.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:46.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:46.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:47.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:47.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:47.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:47.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:48.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:18:48.109+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:48.113694+0000) 2019-10-18T00:18:48.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:48.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:48.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:48.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:49.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:49.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:49.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:49.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:49.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:49.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:50.139 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:18:50.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:50.136272+0000) 2019-10-18T00:18:50.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:50.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:50.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:50.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:51.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:51.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:51.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:51.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:51.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:18:51.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:51.568180+0000) 2019-10-18T00:18:52.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:52.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:52.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:52.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:53.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:53.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:53.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:53.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:54.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:54.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:54.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:54.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:54.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:54.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:18:55.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:55.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:55.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:55.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:56.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:18:56.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:56.145191+0000) 2019-10-18T00:18:56.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:56.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:56.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:56.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:57.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:57.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:57.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:57.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:58.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:18:58.113+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:18:58.113805+0000) 2019-10-18T00:18:58.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:58.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:58.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:58.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:59.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:59.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:18:59.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:18:59.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:18:59.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:18:59.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:00.139 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:19:00.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:00.136447+0000) 2019-10-18T00:19:00.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:00.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:00.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:00.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:01.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:19:01.565+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:01.568230+0000) 2019-10-18T00:19:01.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:01.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:01.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:01.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:02.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:02.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:02.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:02.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:03.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:03.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:03.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:03.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:04.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:04.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:04.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:04.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:04.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:04.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:05.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:05.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:05.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:05.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:06.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:19:06.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:06.145376+0000) 2019-10-18T00:19:06.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:06.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:06.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:06.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:07.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:07.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:07.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:07.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:08.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:19:08.113+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:08.113918+0000) 2019-10-18T00:19:08.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:08.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:08.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:08.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:09.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:09.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:09.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:09.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:09.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:09.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:10.140 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:19:10.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:10.136637+0000) 2019-10-18T00:19:10.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:10.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:10.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:10.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:11.571 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:19:11.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:11.568329+0000) 2019-10-18T00:19:11.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:11.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:11.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:11.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:12.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:12.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:12.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:12.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:13.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:13.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:13.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:13.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:14.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:14.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:14.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:14.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:14.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:14.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:15.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:15.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:15.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:15.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:16.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:19:16.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:16.145557+0000) 2019-10-18T00:19:16.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:16.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:16.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:16.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:17.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:17.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:17.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:17.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:18.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:19:18.112+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:18.114028+0000) 2019-10-18T00:19:18.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:18.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:18.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:18.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:19.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:19.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:19.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:19.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:19.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:19.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:20.140 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:19:20.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:20.136822+0000) 2019-10-18T00:19:20.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:20.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:20.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:20.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:21.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:21.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:21.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:19:21.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:21.568427+0000) 2019-10-18T00:19:22.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:22.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:22.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:22.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:23.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:23.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:23.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:23.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:24.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:24.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:24.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:24.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:24.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:24.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:25.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:25.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:25.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:25.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:26.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:19:26.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:26.145746+0000) 2019-10-18T00:19:26.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:26.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:26.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:26.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:27.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:27.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:27.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:27.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:28.118 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:19:28.112+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:28.114141+0000) 2019-10-18T00:19:28.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:28.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:28.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:28.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:29.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:29.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:29.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:29.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:29.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:29.696+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:30.140 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:19:30.132+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:30.137002+0000) 2019-10-18T00:19:30.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:30.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:30.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:30.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:31.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:19:31.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:31.568529+0000) 2019-10-18T00:19:31.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:31.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:31.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:31.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:32.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:32.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:32.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:32.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:33.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:33.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:33.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:33.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:34.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:34.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:34.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:34.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:34.703 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:34.696+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:35.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:35.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:35.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:35.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:36.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:19:36.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:36.145934+0000) 2019-10-18T00:19:36.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:36.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:36.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:36.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:37.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:37.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:37.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:37.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:38.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:19:38.112+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:38.114252+0000) 2019-10-18T00:19:38.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:38.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:38.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:38.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:39.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:39.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:39.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:39.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:39.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:39.696+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:40.140 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:19:40.131+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:40.137192+0000) 2019-10-18T00:19:40.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:40.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:40.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:40.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:41.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:19:41.564+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:41.568625+0000) 2019-10-18T00:19:41.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:41.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:41.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:41.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:42.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:42.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:42.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:42.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:43.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:43.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:43.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:43.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:44.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:44.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:44.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:44.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:44.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:44.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:45.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:45.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:45.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:45.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:46.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:19:46.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:46.146123+0000) 2019-10-18T00:19:46.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:46.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:46.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:46.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:47.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:47.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:47.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:47.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:48.117 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:19:48.111+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:48.114364+0000) 2019-10-18T00:19:48.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:48.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:48.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:48.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:49.705 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:49.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:49.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:49.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:49.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:49.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:50.140 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:19:50.131+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:50.137354+0000) 2019-10-18T00:19:50.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:50.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:50.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:50.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:51.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:19:51.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:51.568724+0000) 2019-10-18T00:19:51.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:51.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:51.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:51.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:52.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:52.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:52.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:52.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:53.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:53.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:53.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:53.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:54.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:54.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:54.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:54.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:54.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:54.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:55.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:55.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:55.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:55.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:56.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:19:56.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:56.146323+0000) 2019-10-18T00:19:56.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:56.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:56.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:56.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:57.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:57.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:57.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:57.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:58.118 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:19:58.111+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:19:58.114480+0000) 2019-10-18T00:19:58.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:58.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:58.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:58.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:19:59.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:19:59.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:19:59.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:59.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:19:59.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:19:59.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:00.141 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:20:00.131+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:00.137525+0000) 2019-10-18T00:20:00.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:00.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:00.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:00.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:01.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:20:01.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:01.568828+0000) 2019-10-18T00:20:01.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:01.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:01.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:01.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:02.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:02.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:02.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:02.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:03.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:03.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:04.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:04.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:04.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:04.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:04.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:04.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:05.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:05.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:05.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:05.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:06.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:20:06.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:06.146503+0000) 2019-10-18T00:20:06.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:06.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:06.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:06.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:07.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:07.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:07.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:07.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:08.118 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:08.111+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:08.114591+0000) 2019-10-18T00:20:08.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:08.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:08.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:08.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:09.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:09.703+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:09.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:09.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:09.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:09.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:10.141 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:20:10.131+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:10.137714+0000) 2019-10-18T00:20:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:10.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:10.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:10.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:11.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:20:11.563+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:11.568927+0000) 2019-10-18T00:20:11.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:11.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:11.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:11.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:12.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:12.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:13.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:13.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:14.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:14.703+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:14.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:14.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:14.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:14.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:15.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:15.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:15.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:15.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:16.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:20:16.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:16.146668+0000) 2019-10-18T00:20:16.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:16.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:16.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:16.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:17.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:17.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:17.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:17.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:18.120 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:18.111+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:18.114698+0000) 2019-10-18T00:20:18.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:18.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:18.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:18.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:19.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:19.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:19.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:19.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:19.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:19.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:20.141 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:20:20.131+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:20.137952+0000) 2019-10-18T00:20:20.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:20.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:20.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:20.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:21.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:20:21.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:21.569025+0000) 2019-10-18T00:20:21.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:21.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:21.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:21.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:22.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:22.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:22.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:22.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:23.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:23.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:23.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:23.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:24.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:24.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:24.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:24.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:24.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:24.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:25.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:25.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:25.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:25.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:26.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:20:26.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:26.146863+0000) 2019-10-18T00:20:26.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:26.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:26.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:26.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:27.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:27.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:27.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:27.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:28.118 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:28.110+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:28.114807+0000) 2019-10-18T00:20:28.119 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:28.110+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:28.115661+0000) 2019-10-18T00:20:28.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:28.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:28.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:28.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:29.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:29.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:29.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:29.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:29.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:29.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:30.141 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:20:30.135+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:30.138250+0000) 2019-10-18T00:20:30.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:30.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:30.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:30.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:31.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:20:31.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:31.569128+0000) 2019-10-18T00:20:31.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:31.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:31.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:31.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:32.618 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:32.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:32.615021+0000) 2019-10-18T00:20:32.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:32.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:32.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:32.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:33.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:33.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:33.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:33.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:34.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:34.706+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:34.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:34.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:34.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:34.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:35.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:35.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:35.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:35.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:36.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:20:36.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:36.147064+0000) 2019-10-18T00:20:36.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:36.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:36.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:36.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:37.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:37.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:37.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:37.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:38.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:38.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:38.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:38.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:39.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:39.706+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:39.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:39.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:39.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:39.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:40.141 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:20:40.135+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:40.138490+0000) 2019-10-18T00:20:40.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:40.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:40.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:40.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:41.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:20:41.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:41.569231+0000) 2019-10-18T00:20:41.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:41.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:41.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:41.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:42.618 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:42.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:42.615128+0000) 2019-10-18T00:20:42.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:42.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:42.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:42.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:43.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:43.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:43.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:43.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:44.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:44.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:44.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:44.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:44.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:44.706+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:45.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:45.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:45.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:45.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:46.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:20:46.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:46.147247+0000) 2019-10-18T00:20:46.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:46.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:46.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:46.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:47.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:47.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:47.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:47.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:48.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:48.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:48.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:48.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:49.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:49.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:49.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:49.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:49.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:49.706+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:50.142 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:20:50.135+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:50.138728+0000) 2019-10-18T00:20:50.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:50.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:50.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:50.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:51.572 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:20:51.562+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:51.569329+0000) 2019-10-18T00:20:51.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:51.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:51.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:51.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:52.618 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:20:52.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:52.615227+0000) 2019-10-18T00:20:52.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:52.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:52.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:52.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:53.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:53.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:53.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:53.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:54.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:54.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:20:54.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:54.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:54.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:54.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:55.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:55.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:55.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:55.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:56.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:20:56.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:20:56.147439+0000) 2019-10-18T00:20:56.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:56.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:56.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:56.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:57.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:57.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:57.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:57.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:58.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:58.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:58.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:58.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:59.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:59.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:20:59.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:20:59.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:20:59.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:20:59.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:00.142 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:00.134+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:00.138960+0000) 2019-10-18T00:21:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:00.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:00.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:01.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:21:01.561+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:01.569426+0000) 2019-10-18T00:21:01.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:01.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:01.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:01.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:02.618 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:21:02.609+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:02.615327+0000) 2019-10-18T00:21:02.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:02.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:02.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:02.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:03.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:03.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:03.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:03.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:04.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:04.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:04.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:04.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:04.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:04.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:05.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:05.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:05.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:05.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:06.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:21:06.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:06.147625+0000) 2019-10-18T00:21:06.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:06.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:06.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:06.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:07.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:07.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:07.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:07.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:08.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:08.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:08.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:08.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:09.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:09.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:09.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:09.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:10.142 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:10.134+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:10.139102+0000) 2019-10-18T00:21:10.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:10.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:10.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:10.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:11.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:21:11.565+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:11.569533+0000) 2019-10-18T00:21:11.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:11.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:11.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:11.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:12.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:21:12.609+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:12.615425+0000) 2019-10-18T00:21:12.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:12.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:12.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:12.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:13.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:13.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:13.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:13.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:14.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:14.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:14.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:14.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:14.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:14.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:15.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:15.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:15.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:15.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:16.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:21:16.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:16.147827+0000) 2019-10-18T00:21:16.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:16.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:16.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:16.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:17.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:17.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:17.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:17.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:18.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:18.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:18.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:18.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:19.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:19.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:19.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:19.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:19.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:19.717+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:20.142 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:20.134+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:20.139228+0000) 2019-10-18T00:21:20.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:20.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:20.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:20.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:21.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:21:21.569+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:21.569631+0000) 2019-10-18T00:21:21.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:21.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:21.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:21.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:22.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:21:22.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:22.615526+0000) 2019-10-18T00:21:22.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:22.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:22.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:22.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:23.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:23.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:23.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:23.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:24.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:24.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:24.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:24.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:24.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:24.717+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:25.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:25.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:25.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:25.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:26.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:21:26.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:26.148034+0000) 2019-10-18T00:21:26.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:26.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:26.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:26.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:27.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:27.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:27.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:27.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:28.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:28.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:28.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:28.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:29.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:29.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:29.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:29.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:29.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:29.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:30.142 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:30.134+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:30.139363+0000) 2019-10-18T00:21:30.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:30.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:30.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:30.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:31.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:31.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:31.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:31.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:31.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:21:31.568+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:31.569725+0000) 2019-10-18T00:21:32.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:32.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:32.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:32.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:32.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:21:32.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:32.615625+0000) 2019-10-18T00:21:33.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:33.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:33.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:33.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:34.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:34.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:34.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:34.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:34.723 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:34.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:35.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:35.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:35.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:35.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:36.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:21:36.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:36.148252+0000) 2019-10-18T00:21:36.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:36.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:36.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:36.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:37.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:37.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:37.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:37.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:38.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:38.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:38.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:38.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:39.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:39.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:39.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:39.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:39.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:39.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:40.142 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:40.134+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:40.139487+0000) 2019-10-18T00:21:40.144 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:40.134+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:40.140264+0000) 2019-10-18T00:21:40.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:40.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:40.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:40.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:41.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:41.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:41.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:41.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:41.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:21:41.568+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:41.569821+0000) 2019-10-18T00:21:42.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:42.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:42.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:42.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:42.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:21:42.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:42.615719+0000) 2019-10-18T00:21:43.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:43.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:43.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:43.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:44.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:44.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:44.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:44.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:44.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:44.634+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:44.639698+0000) 2019-10-18T00:21:44.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:44.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:45.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:45.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:45.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:45.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:46.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:21:46.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:46.148466+0000) 2019-10-18T00:21:46.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:46.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:46.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:46.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:47.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:47.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:47.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:47.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:48.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:48.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:48.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:48.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:49.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:49.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:49.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:49.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:49.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:49.720+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:50.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:50.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:50.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:50.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:51.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:51.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:51.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:51.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:51.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:21:51.568+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:51.569919+0000) 2019-10-18T00:21:52.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:52.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:11:15.904064+0000 (oldest deadline 2019-10-18T00:11:35.904064+0000) 2019-10-18T00:21:52.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:52.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:52.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:21:52.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:52.615819+0000) 2019-10-18T00:21:53.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:53.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:54.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:54.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:54.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:21:54.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:54.639810+0000) 2019-10-18T00:21:54.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:54.720+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:21:55.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:55.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:56.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:21:56.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:21:56.148641+0000) 2019-10-18T00:21:56.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:56.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:57.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:57.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:58.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:58.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:59.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:21:59.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:21:59.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:21:59.720+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:00.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:00.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:01.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:01.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:01.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:01.568+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:01.570018+0000) 2019-10-18T00:22:02.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:02.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:02.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:22:02.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:02.615923+0000) 2019-10-18T00:22:03.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:03.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:04.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:04.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:04.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:22:04.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:04.639973+0000) 2019-10-18T00:22:04.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:04.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:05.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:05.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:06.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:22:06.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:06.148832+0000) 2019-10-18T00:22:06.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:06.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:07.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:07.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:08.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:08.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:09.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:09.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:09.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:09.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:10.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:10.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:11.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:11.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:11.575 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:11.567+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:11.570113+0000) 2019-10-18T00:22:12.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:12.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:12.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:22:12.611+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:12.616023+0000) 2019-10-18T00:22:13.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:13.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:14.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:14.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:14.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:14.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:14.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:22:14.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:14.640185+0000) 2019-10-18T00:22:14.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:14.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:15.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:15.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:15.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:15.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:16.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:22:16.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:16.149012+0000) 2019-10-18T00:22:16.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:16.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:16.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:16.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:17.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:17.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:17.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:17.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:18.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:18.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:18.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:18.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:19.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:19.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:19.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:19.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:19.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:19.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:20.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:20.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:20.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:20.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:21.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:21.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:21.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:21.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:21.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:21.567+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:21.570206+0000) 2019-10-18T00:22:22.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:22.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:22.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:22.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:22.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:22:22.611+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:22.616125+0000) 2019-10-18T00:22:23.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:23.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:23.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:23.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:24.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:24.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:24.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:24.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:24.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:22:24.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:24.640367+0000) 2019-10-18T00:22:24.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:24.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:25.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:25.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:26.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:22:26.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:26.149230+0000) 2019-10-18T00:22:26.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:26.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:26.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:26.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:27.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:27.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:27.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:27.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:28.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:28.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:28.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:28.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:29.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:29.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:29.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:29.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:29.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:29.727+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:30.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:30.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:30.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:30.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:31.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:31.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:31.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:31.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:31.573 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:31.567+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:31.570310+0000) 2019-10-18T00:22:32.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:32.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:32.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:32.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:32.619 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:22:32.611+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:32.616223+0000) 2019-10-18T00:22:33.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:33.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:33.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:33.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:34.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:34.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:34.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:34.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:34.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:22:34.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:34.640547+0000) 2019-10-18T00:22:34.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:34.727+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:35.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:35.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:35.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:35.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:36.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:22:36.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:36.149433+0000) 2019-10-18T00:22:36.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:36.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:36.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:36.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:37.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:37.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:37.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:37.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:38.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:38.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:38.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:38.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:39.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:39.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:39.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:39.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:39.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:39.727+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:40.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:40.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:40.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:40.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:41.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:41.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:41.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:41.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:41.574 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:41.566+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:41.570409+0000) 2019-10-18T00:22:41.575 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:41.566+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:41.571347+0000) 2019-10-18T00:22:42.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:42.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:42.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:42.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:42.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:22:42.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:42.616322+0000) 2019-10-18T00:22:43.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:43.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:43.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:43.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:44.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:44.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:44.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:44.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:44.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:22:44.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:44.640738+0000) 2019-10-18T00:22:44.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:44.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:45.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:45.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:45.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:45.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:46.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:46.066+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:46.070613+0000) 2019-10-18T00:22:46.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:22:46.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:46.149621+0000) 2019-10-18T00:22:46.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:46.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:46.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:46.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:47.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:47.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:47.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:47.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:48.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:48.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:48.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:48.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:49.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:49.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:49.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:49.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:49.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:49.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:50.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:50.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:50.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:50.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:51.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:51.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:51.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:51.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:52.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:52.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:52.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:52.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:52.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:22:52.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:52.616422+0000) 2019-10-18T00:22:53.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:53.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:53.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:53.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:54.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:54.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:54.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:54.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:54.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:22:54.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:54.640923+0000) 2019-10-18T00:22:54.737 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:54.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:22:55.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:55.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:55.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:55.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:56.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:22:56.066+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:56.070712+0000) 2019-10-18T00:22:56.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:22:56.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:22:56.149833+0000) 2019-10-18T00:22:56.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:56.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:56.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:56.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:57.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:57.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:57.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:57.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:58.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:58.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:58.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:58.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:59.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:59.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:22:59.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:22:59.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:22:59.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:22:59.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:00.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:00.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:00.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:00.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:01.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:01.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:01.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:01.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:02.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:02.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:02.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:23:02.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:02.616524+0000) 2019-10-18T00:23:03.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:03.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:03.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:03.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:04.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:04.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:04.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:04.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:04.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:23:04.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:04.641103+0000) 2019-10-18T00:23:04.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:04.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:05.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:05.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:05.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:05.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:06.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:05.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:06.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:05.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:06.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:23:06.066+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:06.070814+0000) 2019-10-18T00:23:06.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:23:06.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:06.150056+0000) 2019-10-18T00:23:07.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:07.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:07.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:07.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:08.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:08.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:08.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:08.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:09.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:09.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:09.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:09.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:09.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:09.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:10.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:10.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:10.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:10.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:11.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:11.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:11.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:11.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:12.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:12.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:12.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:12.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:12.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:23:12.610+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:12.616621+0000) 2019-10-18T00:23:13.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:13.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:13.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:13.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:14.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:14.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:14.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:14.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:14.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:23:14.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:14.641287+0000) 2019-10-18T00:23:14.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:14.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:15.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:15.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:15.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:15.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:16.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:23:16.065+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:16.070925+0000) 2019-10-18T00:23:16.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:16.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:16.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:16.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:16.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:23:16.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:16.150257+0000) 2019-10-18T00:23:17.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:17.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:17.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:17.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:18.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:18.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:18.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:18.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:19.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:19.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:19.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:19.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:19.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:19.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:20.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:20.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:20.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:20.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:21.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:21.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:21.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:21.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:22.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:22.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:22.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:22.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:22.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:23:22.609+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:22.616727+0000) 2019-10-18T00:23:23.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:23.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:23.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:23.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:24.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:24.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:24.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:24.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:24.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:23:24.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:24.641505+0000) 2019-10-18T00:23:24.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:24.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:25.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:25.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:26.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:23:26.065+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:26.071025+0000) 2019-10-18T00:23:26.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:26.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:26.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:26.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:26.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:23:26.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:26.150410+0000) 2019-10-18T00:23:27.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:27.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:27.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:27.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:28.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:28.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:28.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:28.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:29.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:29.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:29.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:29.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:29.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:29.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:30.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:30.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:30.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:30.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:31.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:31.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:31.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:31.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:32.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:32.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:32.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:32.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:32.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:23:32.609+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:32.616835+0000) 2019-10-18T00:23:33.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:33.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:33.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:33.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:34.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:34.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:34.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:34.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:34.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:23:34.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:34.641724+0000) 2019-10-18T00:23:34.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:34.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:35.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:35.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:35.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:35.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:36.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:23:36.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:36.071125+0000) 2019-10-18T00:23:36.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:36.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:36.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:36.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:36.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:23:36.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:36.150587+0000) 2019-10-18T00:23:37.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:37.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:37.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:37.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:38.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:38.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:38.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:38.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:39.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:39.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:39.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:39.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:39.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:39.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:40.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:40.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:40.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:40.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:41.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:41.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:41.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:41.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:42.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:42.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:42.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:42.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:42.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:23:42.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:42.616931+0000) 2019-10-18T00:23:43.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:43.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:43.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:43.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:44.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:44.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:44.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:44.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:44.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:23:44.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:44.641935+0000) 2019-10-18T00:23:44.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:44.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:45.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:45.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:45.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:45.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:46.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:46.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:46.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:46.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:46.074 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:23:46.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:46.071229+0000) 2019-10-18T00:23:46.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:23:46.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:46.150755+0000) 2019-10-18T00:23:47.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:47.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:47.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:47.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:48.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:48.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:48.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:48.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:48.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:48.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:49.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:49.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:49.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:49.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:49.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:49.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:50.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:50.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:50.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:50.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:51.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:51.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:51.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:51.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:52.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:23:52.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:52.617032+0000) 2019-10-18T00:23:52.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:52.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:52.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:52.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:53.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:53.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:53.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:53.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:54.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:23:54.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:54.642124+0000) 2019-10-18T00:23:54.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:54.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:23:54.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:54.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:54.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:54.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:56.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:56.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:56.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:56.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:56.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:23:56.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:56.071332+0000) 2019-10-18T00:23:56.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:23:56.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:23:56.150942+0000) 2019-10-18T00:23:57.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:57.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:57.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:57.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:58.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:57.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:58.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:57.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:59.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:59.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:23:59.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:23:59.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:23:59.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:23:59.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:00.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:00.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:00.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:00.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:01.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:01.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:01.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:01.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:02.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:02.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:02.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:02.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:02.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:24:02.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:02.617135+0000) 2019-10-18T00:24:03.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:03.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:03.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:03.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:04.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:04.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:04.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:04.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:04.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:24:04.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:04.642301+0000) 2019-10-18T00:24:04.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:04.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:05.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:05.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:05.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:05.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:06.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:24:06.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:06.071433+0000) 2019-10-18T00:24:06.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:06.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:06.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:06.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:06.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:24:06.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:06.151122+0000) 2019-10-18T00:24:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:07.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:07.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:08.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:08.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:08.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:08.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:09.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:09.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:09.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:09.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:09.750 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:09.744+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:10.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:10.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:10.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:10.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:11.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:11.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:11.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:11.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:12.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:12.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:12.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:12.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:12.620 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:24:12.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:12.617243+0000) 2019-10-18T00:24:13.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:13.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:13.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:13.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:14.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:14.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:14.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:14.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:14.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:24:14.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:14.642485+0000) 2019-10-18T00:24:14.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:14.744+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:15.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:15.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:15.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:15.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:16.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:24:16.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:16.071538+0000) 2019-10-18T00:24:16.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:16.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:16.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:16.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:16.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:24:16.148+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:16.151294+0000) 2019-10-18T00:24:17.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:17.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:17.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:17.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:18.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:18.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:18.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:18.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:19.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:19.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:19.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:19.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:19.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:19.744+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:20.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:20.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:20.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:20.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:21.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:21.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:21.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:21.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:22.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:22.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:22.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:22.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:24:22.612+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:22.617327+0000) 2019-10-18T00:24:23.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:23.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:23.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:23.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:24.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:24.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:24.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:24.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:24.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:24:24.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:24.642701+0000) 2019-10-18T00:24:24.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:24.744+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:25.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:25.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:25.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:25.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:26.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:24:26.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:26.071641+0000) 2019-10-18T00:24:26.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:24:26.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:26.151482+0000) 2019-10-18T00:24:26.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:26.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:26.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:26.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:27.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:27.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:27.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:27.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:28.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:28.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:28.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:28.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:29.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:29.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:29.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:29.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:29.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:29.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:30.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:30.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:30.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:30.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:31.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:31.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:31.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:31.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:32.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:32.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:32.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:32.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:32.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:24:32.611+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:32.617426+0000) 2019-10-18T00:24:33.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:33.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:33.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:33.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:34.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:34.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:34.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:34.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:34.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:24:34.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:34.642888+0000) 2019-10-18T00:24:34.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:34.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:35.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:35.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:35.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:35.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:36.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:24:36.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:36.071739+0000) 2019-10-18T00:24:36.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:24:36.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:36.151659+0000) 2019-10-18T00:24:36.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:36.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:36.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:36.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:37.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:37.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:37.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:37.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:38.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:38.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:38.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:38.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:39.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:39.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:39.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:39.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:39.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:39.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:40.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:40.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:40.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:40.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:41.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:41.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:41.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:41.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:42.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:42.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:42.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:42.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:42.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:24:42.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:42.617542+0000) 2019-10-18T00:24:43.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:43.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:43.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:43.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:44.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:44.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:44.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:44.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:44.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:24:44.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:44.643106+0000) 2019-10-18T00:24:44.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:44.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:45.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:45.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:45.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:45.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:46.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:24:46.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:46.071840+0000) 2019-10-18T00:24:46.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:24:46.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:46.151849+0000) 2019-10-18T00:24:46.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:46.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:46.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:46.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:47.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:47.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:47.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:47.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:48.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:48.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:48.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:48.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:49.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:49.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:49.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:49.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:49.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:49.751+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:50.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:50.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:50.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:50.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:51.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:51.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:51.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:51.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:52.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:52.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:52.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:52.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:52.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:24:52.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:52.617646+0000) 2019-10-18T00:24:53.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:53.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:53.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:53.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:54.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:54.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:54.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:54.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:54.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:24:54.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:54.643292+0000) 2019-10-18T00:24:54.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:54.751+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:24:55.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:55.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:55.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:55.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:56.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:24:56.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:56.071933+0000) 2019-10-18T00:24:56.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:24:56.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:24:56.151990+0000) 2019-10-18T00:24:56.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:56.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:56.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:56.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:57.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:57.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:57.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:57.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:58.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:58.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:58.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:58.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:59.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:59.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:24:59.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:24:59.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:24:59.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:24:59.751+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:00.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:00.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:01.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:01.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:01.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:01.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:02.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:02.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:02.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:02.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:02.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:25:02.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:02.617742+0000) 2019-10-18T00:25:03.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:03.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:03.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:03.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:04.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:04.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:04.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:04.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:04.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:25:04.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:04.643500+0000) 2019-10-18T00:25:04.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:04.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:05.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:05.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:05.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:05.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:06.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:25:06.066+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:06.072035+0000) 2019-10-18T00:25:06.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:25:06.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:06.152169+0000) 2019-10-18T00:25:06.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:06.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:06.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:06.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:07.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:07.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:07.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:07.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:08.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:08.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:08.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:08.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:09.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:09.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:09.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:09.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:09.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:09.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:10.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:10.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:10.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:10.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:11.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:11.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:11.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:11.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:12.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:12.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:12.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:12.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:12.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:25:12.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:12.617847+0000) 2019-10-18T00:25:13.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:13.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:13.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:13.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:14.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:14.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:14.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:14.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:14.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:25:14.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:14.643695+0000) 2019-10-18T00:25:14.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:14.754+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:15.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:15.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:15.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:15.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:16.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:25:16.066+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:16.072139+0000) 2019-10-18T00:25:16.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:25:16.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:16.152354+0000) 2019-10-18T00:25:16.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:16.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:16.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:16.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:17.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:17.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:17.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:17.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:18.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:18.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:18.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:18.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:19.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:19.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:19.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:19.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:19.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:19.754+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:20.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:20.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:20.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:20.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:21.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:21.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:21.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:21.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:22.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:22.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:22.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:22.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:22.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:25:22.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:22.617948+0000) 2019-10-18T00:25:23.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:23.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:23.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:23.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:24.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:24.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:24.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:24.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:24.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:25:24.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:24.643879+0000) 2019-10-18T00:25:24.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:24.754+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:25.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:25.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:25.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:25.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:26.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:25:26.066+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:26.072243+0000) 2019-10-18T00:25:26.156 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:25:26.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:26.152545+0000) 2019-10-18T00:25:26.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:26.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:26.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:26.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:27.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:27.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:27.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:27.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:28.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:28.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:28.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:28.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:29.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:29.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:29.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:29.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:29.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:29.754+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:30.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:30.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:30.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:30.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:31.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:31.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:31.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:31.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:32.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:32.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:32.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:32.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:32.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:25:32.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:32.618053+0000) 2019-10-18T00:25:33.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:33.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:33.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:33.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:34.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:34.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:34.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:34.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:34.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:25:34.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:34.644060+0000) 2019-10-18T00:25:34.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:34.754+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:35.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:35.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:35.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:35.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:36.075 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:25:36.065+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:36.072346+0000) 2019-10-18T00:25:36.156 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:25:36.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:36.152754+0000) 2019-10-18T00:25:36.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:36.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:36.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:36.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:37.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:37.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:37.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:37.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:38.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:38.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:38.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:38.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:39.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:39.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:39.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:39.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:39.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:39.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:40.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:40.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:40.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:40.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:41.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:41.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:41.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:41.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:42.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:42.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:42.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:42.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:42.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:25:42.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:42.618157+0000) 2019-10-18T00:25:43.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:43.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:43.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:43.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:44.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:44.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:44.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:44.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:44.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:25:44.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:44.644253+0000) 2019-10-18T00:25:44.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:44.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:45.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:45.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:45.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:45.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:46.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:25:46.065+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:46.072451+0000) 2019-10-18T00:25:46.156 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:25:46.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:46.152955+0000) 2019-10-18T00:25:46.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:46.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:46.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:46.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:47.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:47.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:47.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:47.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:48.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:48.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:48.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:48.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:49.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:49.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:49.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:49.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:49.767 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:49.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:50.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:50.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:50.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:50.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:51.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:51.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:51.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:51.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:52.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:52.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:52.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:52.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:52.621 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:25:52.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:52.618271+0000) 2019-10-18T00:25:53.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:53.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:53.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:53.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:54.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:54.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:54.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:54.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:54.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:25:54.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:54.644435+0000) 2019-10-18T00:25:54.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:54.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:25:55.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:55.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:55.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:55.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:56.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:25:56.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:56.072555+0000) 2019-10-18T00:25:56.156 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:25:56.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:25:56.153133+0000) 2019-10-18T00:25:56.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:56.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:56.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:56.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:57.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:57.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:57.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:57.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:58.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:58.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:59.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:59.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:25:59.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:25:59.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:25:59.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:25:59.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:00.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:00.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:00.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:00.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:01.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:01.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:01.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:01.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:02.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:02.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:02.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:02.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:02.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:26:02.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:02.618376+0000) 2019-10-18T00:26:03.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:03.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:03.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:03.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:04.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:04.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:04.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:04.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:04.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:26:04.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:04.644622+0000) 2019-10-18T00:26:04.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:04.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:05.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:05.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:05.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:05.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:06.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:26:06.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:06.072656+0000) 2019-10-18T00:26:06.156 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:26:06.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:06.153340+0000) 2019-10-18T00:26:06.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:06.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:06.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:06.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:07.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:07.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:07.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:07.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:08.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:08.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:08.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:08.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:09.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:09.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:09.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:09.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:09.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:09.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:10.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:10.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:10.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:10.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:11.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:11.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:12.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:12.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:12.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:12.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:12.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:26:12.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:12.618484+0000) 2019-10-18T00:26:13.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:13.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:13.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:13.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:14.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:14.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:14.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:14.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:14.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:26:14.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:14.644802+0000) 2019-10-18T00:26:14.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:14.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:15.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:15.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:15.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:15.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:16.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:26:16.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:16.072758+0000) 2019-10-18T00:26:16.157 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:26:16.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:16.153543+0000) 2019-10-18T00:26:16.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:16.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:16.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:16.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:17.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:17.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:17.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:17.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:18.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:18.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:18.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:18.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:19.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:19.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:19.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:19.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:19.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:19.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:20.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:20.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:21.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:21.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:21.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:21.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:22.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:26:22.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:22.618588+0000) 2019-10-18T00:26:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:22.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:22.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:23.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:23.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:23.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:23.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:24.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:26:24.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:24.644987+0000) 2019-10-18T00:26:24.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:24.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:24.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:24.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:24.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:24.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:25.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:25.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:25.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:25.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:26.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:26:26.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:26.072856+0000) 2019-10-18T00:26:26.157 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:26:26.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:26.153725+0000) 2019-10-18T00:26:26.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:26.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:26.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:26.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:27.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:27.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:27.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:27.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:28.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:28.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:28.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:28.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:29.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:29.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:29.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:29.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:29.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:29.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:30.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:30.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:30.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:30.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:31.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:31.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:31.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:31.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:32.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:26:32.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:32.618696+0000) 2019-10-18T00:26:32.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:32.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:32.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:32.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:33.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:33.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:33.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:33.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:34.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:26:34.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:34.645181+0000) 2019-10-18T00:26:34.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:34.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:34.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:34.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:34.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:34.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:35.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:35.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:35.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:35.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:36.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:26:36.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:36.072955+0000) 2019-10-18T00:26:36.157 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:26:36.150+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:36.153920+0000) 2019-10-18T00:26:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:36.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:36.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:37.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:37.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:37.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:37.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:38.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:38.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:38.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:38.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:39.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:39.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:39.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:39.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:39.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:39.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:40.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:40.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:40.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:40.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:41.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:41.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:41.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:41.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:42.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:26:42.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:42.618798+0000) 2019-10-18T00:26:42.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:42.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:42.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:42.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:43.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:43.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:43.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:43.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:44.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:26:44.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:44.645396+0000) 2019-10-18T00:26:44.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:44.768+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:44.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:44.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:44.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:44.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:45.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:45.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:45.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:45.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:46.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:26:46.068+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:46.073055+0000) 2019-10-18T00:26:46.157 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:26:46.149+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:46.154106+0000) 2019-10-18T00:26:46.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:46.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:46.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:46.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:47.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:47.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:47.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:47.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:48.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:48.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:48.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:48.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:49.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:49.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:49.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:49.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:49.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:49.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:50.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:50.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:50.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:50.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:51.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:51.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:51.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:51.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:52.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:26:52.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:52.618898+0000) 2019-10-18T00:26:52.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:52.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:52.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:52.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:53.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:53.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:53.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:53.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:54.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:26:54.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:54.645585+0000) 2019-10-18T00:26:54.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:54.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:54.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:54.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:54.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:54.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:55.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:55.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:56.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:26:56.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:56.073158+0000) 2019-10-18T00:26:56.157 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:26:56.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:26:56.154270+0000) 2019-10-18T00:26:56.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:56.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:56.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:56.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:57.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:57.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:57.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:57.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:58.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:58.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:58.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:58.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:26:59.779 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:26:59.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:26:59.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:59.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:26:59.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:26:59.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:00.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:00.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:00.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:00.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:01.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:01.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:01.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:01.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:02.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:27:02.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:02.619000+0000) 2019-10-18T00:27:02.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:02.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:02.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:02.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:03.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:03.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:03.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:03.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:04.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:27:04.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:04.645728+0000) 2019-10-18T00:27:04.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:04.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:04.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:04.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:04.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:04.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:05.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:05.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:05.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:05.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:06.076 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:27:06.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:06.073258+0000) 2019-10-18T00:27:06.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:27:06.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:06.154464+0000) 2019-10-18T00:27:06.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:06.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:06.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:06.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:07.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:07.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:07.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:07.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:08.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:08.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:08.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:08.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:09.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:09.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:09.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:09.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:09.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:09.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:10.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:10.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:10.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:10.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:11.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:11.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:11.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:11.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:12.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:27:12.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:12.619108+0000) 2019-10-18T00:27:12.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:12.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:12.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:12.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:13.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:13.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:13.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:13.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:14.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:27:14.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:14.645940+0000) 2019-10-18T00:27:14.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:14.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:14.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:14.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:14.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:14.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:15.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:15.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:15.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:15.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:16.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:27:16.067+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:16.073365+0000) 2019-10-18T00:27:16.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:27:16.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:16.154636+0000) 2019-10-18T00:27:16.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:16.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:16.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:16.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:17.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:17.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:17.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:17.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:18.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:18.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:18.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:18.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:19.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:19.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:19.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:19.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:19.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:19.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:20.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:20.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:20.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:20.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:21.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:21.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:21.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:21.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:22.622 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:27:22.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:22.619216+0000) 2019-10-18T00:27:22.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:22.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:22.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:22.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:23.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:23.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:23.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:23.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:24.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:27:24.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:24.646156+0000) 2019-10-18T00:27:24.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:24.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:24.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:24.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:24.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:24.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:25.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:25.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:25.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:25.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:26.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:27:26.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:26.073461+0000) 2019-10-18T00:27:26.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:27:26.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:26.154797+0000) 2019-10-18T00:27:26.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:26.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:26.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:26.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:27.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:27.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:27.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:27.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:28.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:28.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:28.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:28.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:29.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:29.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:29.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:29.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:29.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:29.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:30.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:30.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:30.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:30.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:31.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:31.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:31.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:31.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:32.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:27:32.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:32.619325+0000) 2019-10-18T00:27:32.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:32.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:32.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:32.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:33.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:33.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:33.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:33.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:34.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:27:34.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:34.646359+0000) 2019-10-18T00:27:34.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:34.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:34.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:34.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:34.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:34.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:35.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:35.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:35.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:35.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:36.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:27:36.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:36.073690+0000) 2019-10-18T00:27:36.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:27:36.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:36.155010+0000) 2019-10-18T00:27:36.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:36.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:36.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:36.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:37.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:37.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:37.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:37.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:38.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:38.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:38.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:38.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:39.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:39.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:39.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:39.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:39.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:39.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:40.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:40.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:40.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:40.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:41.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:41.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:41.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:41.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:42.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:42.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:42.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:42.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:42.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:27:42.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:42.619434+0000) 2019-10-18T00:27:43.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:43.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:43.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:43.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:44.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:44.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:44.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:44.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:44.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:27:44.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:44.646546+0000) 2019-10-18T00:27:44.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:44.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:45.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:45.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:45.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:45.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:46.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:27:46.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:46.073792+0000) 2019-10-18T00:27:46.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:27:46.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:46.155203+0000) 2019-10-18T00:27:46.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:46.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:46.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:46.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:47.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:47.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:47.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:47.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:48.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:48.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:48.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:48.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:49.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:49.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:49.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:49.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:49.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:50.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:50.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:50.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:50.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:51.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:51.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:51.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:51.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:52.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:52.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:52.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:52.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:52.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:27:52.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:52.619543+0000) 2019-10-18T00:27:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:53.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:53.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:54.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:54.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:54.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:54.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:54.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:27:54.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:54.646763+0000) 2019-10-18T00:27:54.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:54.778+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:27:55.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:55.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:55.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:55.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:56.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:27:56.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:56.073889+0000) 2019-10-18T00:27:56.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:27:56.153+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:27:56.155403+0000) 2019-10-18T00:27:56.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:56.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:56.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:56.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:57.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:57.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:57.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:57.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:58.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:58.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:58.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:58.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:59.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:59.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:27:59.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:27:59.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:27:59.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:27:59.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:00.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:00.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:00.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:00.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:01.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:01.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:01.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:01.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:02.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:28:02.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:02.619648+0000) 2019-10-18T00:28:02.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:02.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:02.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:02.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:03.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:03.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:03.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:03.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:04.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:04.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:04.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:04.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:04.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:28:04.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:04.646944+0000) 2019-10-18T00:28:04.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:04.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:05.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:05.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:05.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:05.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:06.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:28:06.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:06.073988+0000) 2019-10-18T00:28:06.159 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:28:06.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:06.155581+0000) 2019-10-18T00:28:06.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:06.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:06.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:06.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:07.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:07.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:07.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:07.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:08.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:08.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:08.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:08.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:09.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:09.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:09.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:09.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:09.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:09.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:10.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:10.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:10.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:10.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:11.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:11.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:11.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:11.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:12.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:28:12.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:12.619758+0000) 2019-10-18T00:28:12.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:12.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:12.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:12.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:13.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:13.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:13.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:13.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:14.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:28:14.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:14.647130+0000) 2019-10-18T00:28:14.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:14.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:14.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:14.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:14.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:14.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:15.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:15.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:15.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:15.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:16.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:28:16.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:16.074093+0000) 2019-10-18T00:28:16.159 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:28:16.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:16.155788+0000) 2019-10-18T00:28:16.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:16.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:16.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:16.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:17.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:17.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:17.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:17.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:18.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:18.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:18.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:18.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:19.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:19.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:19.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:19.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:19.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:19.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:20.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:20.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:20.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:20.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:21.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:21.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:21.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:21.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:22.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:28:22.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:22.619865+0000) 2019-10-18T00:28:22.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:22.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:22.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:22.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:23.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:23.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:23.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:23.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:24.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:28:24.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:24.647316+0000) 2019-10-18T00:28:24.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:24.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:24.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:24.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:24.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:24.789+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:25.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:25.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:25.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:25.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:26.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:28:26.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:26.074190+0000) 2019-10-18T00:28:26.159 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:28:26.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:26.155964+0000) 2019-10-18T00:28:26.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:26.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:26.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:26.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:27.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:27.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:27.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:27.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:28.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:28.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:28.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:28.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:29.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:29.789+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:29.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:29.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:29.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:29.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:30.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:30.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:30.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:30.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:31.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:31.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:31.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:31.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:32.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:28:32.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:32.619970+0000) 2019-10-18T00:28:32.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:32.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:32.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:32.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:33.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:33.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:33.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:33.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:34.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:28:34.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:34.647497+0000) 2019-10-18T00:28:34.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:34.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:34.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:34.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:34.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:34.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:35.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:35.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:35.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:35.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:36.077 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:28:36.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:36.074288+0000) 2019-10-18T00:28:36.159 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:28:36.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:36.156154+0000) 2019-10-18T00:28:36.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:36.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:36.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:36.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:37.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:37.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:37.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:37.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:38.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:38.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:38.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:38.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:39.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:39.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:39.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:39.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:39.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:39.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:40.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:40.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:40.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:40.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:41.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:41.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:41.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:41.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:42.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:28:42.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:42.620071+0000) 2019-10-18T00:28:42.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:42.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:42.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:42.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:43.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:43.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:43.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:43.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:44.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:28:44.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:44.647679+0000) 2019-10-18T00:28:44.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:44.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:44.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:44.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:44.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:44.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:45.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:45.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:45.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:45.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:46.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:28:46.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:46.074402+0000) 2019-10-18T00:28:46.159 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:28:46.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:46.156296+0000) 2019-10-18T00:28:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:46.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:46.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:47.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:47.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:47.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:47.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:48.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:48.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:48.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:48.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:49.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:49.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:49.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:49.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:49.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:49.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:50.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:50.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:50.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:50.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:51.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:51.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:51.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:51.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:52.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:28:52.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:52.620171+0000) 2019-10-18T00:28:52.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:52.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:52.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:52.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:53.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:53.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:53.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:53.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:54.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:28:54.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:54.647864+0000) 2019-10-18T00:28:54.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:54.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:54.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:54.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:54.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:54.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:28:55.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:55.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:55.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:55.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:56.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:28:56.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:56.074507+0000) 2019-10-18T00:28:56.160 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:28:56.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:28:56.156505+0000) 2019-10-18T00:28:56.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:56.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:56.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:56.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:57.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:57.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:57.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:57.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:58.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:58.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:58.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:58.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:59.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:28:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:28:59.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:28:59.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:28:59.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:00.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:00.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:00.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:00.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:01.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:01.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:01.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:01.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:02.623 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:29:02.616+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:02.620277+0000) 2019-10-18T00:29:02.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:02.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:02.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:02.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:03.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:03.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:03.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:03.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:04.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:04.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:04.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:04.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:04.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:29:04.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:04.648051+0000) 2019-10-18T00:29:04.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:04.792+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:05.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:05.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:05.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:05.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:06.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:29:06.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:06.074604+0000) 2019-10-18T00:29:06.160 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:29:06.152+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:06.156695+0000) 2019-10-18T00:29:06.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:06.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:06.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:06.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:07.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:07.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:07.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:07.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:08.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:08.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:08.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:08.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:09.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:09.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:09.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:09.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:09.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:09.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:10.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:10.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:10.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:10.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:11.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:11.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:11.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:11.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:12.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:29:12.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:12.620383+0000) 2019-10-18T00:29:12.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:12.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:12.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:12.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:13.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:13.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:13.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:13.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:14.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:14.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:14.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:14.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:14.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:29:14.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:14.648232+0000) 2019-10-18T00:29:14.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:14.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:15.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:15.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:15.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:15.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:16.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:29:16.071+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:16.074700+0000) 2019-10-18T00:29:16.160 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:29:16.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:16.156879+0000) 2019-10-18T00:29:16.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:16.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:16.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:16.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:17.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:17.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:17.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:17.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:18.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:18.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:18.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:18.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:19.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:19.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:19.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:19.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:19.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:19.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:20.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:20.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:20.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:20.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:21.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:21.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:21.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:21.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:22.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:22.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:22.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:22.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:22.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:29:22.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:22.620489+0000) 2019-10-18T00:29:23.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:23.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:23.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:23.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:24.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:24.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:24.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:29:24.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:24.648420+0000) 2019-10-18T00:29:24.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:24.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:25.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:25.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:25.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:25.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:26.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:29:26.071+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:26.074796+0000) 2019-10-18T00:29:26.160 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:29:26.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:26.157085+0000) 2019-10-18T00:29:26.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:26.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:26.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:26.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:27.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:27.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:27.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:27.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:28.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:28.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:28.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:28.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:29.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:29.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:29.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:29.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:29.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:29.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:30.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:30.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:30.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:30.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:31.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:31.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:31.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:31.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:32.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:32.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:32.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:32.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:32.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:29:32.615+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:32.620597+0000) 2019-10-18T00:29:33.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:33.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:33.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:33.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:34.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:29:34.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:34.648603+0000) 2019-10-18T00:29:34.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:34.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:34.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:34.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:34.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:34.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:35.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:35.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:35.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:35.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:36.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:29:36.071+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:36.074894+0000) 2019-10-18T00:29:36.160 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:29:36.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:36.157273+0000) 2019-10-18T00:29:36.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:36.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:36.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:36.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:37.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:37.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:37.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:37.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:38.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:38.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:38.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:38.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:39.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:39.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:39.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:39.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:39.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:39.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:40.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:40.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:40.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:40.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:41.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:41.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:41.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:41.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:42.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:29:42.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:42.620708+0000) 2019-10-18T00:29:42.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:42.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:42.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:42.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:43.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:43.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:43.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:43.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:44.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:29:44.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:44.648786+0000) 2019-10-18T00:29:44.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:44.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:44.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:44.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:44.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:44.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:45.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:45.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:45.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:45.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:46.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:29:46.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:46.074994+0000) 2019-10-18T00:29:46.161 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:29:46.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:46.157457+0000) 2019-10-18T00:29:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:46.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:46.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:47.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:47.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:48.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:48.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:48.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:48.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:49.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:49.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:49.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:49.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:49.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:49.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:50.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:50.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:50.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:50.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:51.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:51.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:51.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:51.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:52.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:29:52.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:52.620818+0000) 2019-10-18T00:29:52.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:52.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:52.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:52.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:53.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:53.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:53.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:53.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:54.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:29:54.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:54.648971+0000) 2019-10-18T00:29:54.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:54.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:54.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:54.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:54.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:54.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:29:55.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:55.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:55.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:55.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:56.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:29:56.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:56.075094+0000) 2019-10-18T00:29:56.161 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:29:56.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:29:56.157645+0000) 2019-10-18T00:29:56.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:56.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:56.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:56.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:57.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:57.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:57.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:57.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:58.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:58.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:58.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:58.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:59.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:59.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:29:59.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:29:59.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:29:59.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:29:59.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:00.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:00.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:00.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:00.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:01.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:01.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:01.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:01.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:02.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:30:02.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:02.620922+0000) 2019-10-18T00:30:02.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:02.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:02.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:02.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:03.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:03.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:03.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:03.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:04.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:30:04.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:04.649157+0000) 2019-10-18T00:30:04.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:04.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:04.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:04.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:04.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:04.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:05.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:05.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:05.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:05.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:06.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:30:06.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:06.075192+0000) 2019-10-18T00:30:06.161 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:30:06.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:06.157784+0000) 2019-10-18T00:30:06.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:06.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:06.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:06.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:07.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:07.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:07.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:07.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:08.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:08.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:08.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:08.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:09.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:09.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:09.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:09.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:10.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:10.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:10.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:10.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:11.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:11.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:11.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:11.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:12.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:30:12.614+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:12.621030+0000) 2019-10-18T00:30:12.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:12.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:12.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:12.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:13.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:13.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:13.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:13.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:14.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:30:14.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:14.649339+0000) 2019-10-18T00:30:14.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:14.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:14.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:14.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:14.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:14.802+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:15.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:15.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:15.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:15.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:16.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:30:16.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:16.075299+0000) 2019-10-18T00:30:16.161 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:30:16.151+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:16.157969+0000) 2019-10-18T00:30:16.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:16.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:16.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:16.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:17.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:17.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:17.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:17.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:18.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:18.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:18.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:18.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:19.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:19.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:19.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:19.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:19.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:19.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:20.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:20.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:20.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:20.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:21.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:21.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:21.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:21.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:22.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:22.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:22.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:22.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:22.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:30:22.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:22.621134+0000) 2019-10-18T00:30:23.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:23.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:23.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:23.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:24.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:24.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:24.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:24.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:24.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:30:24.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:24.649581+0000) 2019-10-18T00:30:24.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:24.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:25.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:25.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:25.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:25.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:26.078 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:30:26.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:26.075398+0000) 2019-10-18T00:30:26.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:30:26.150+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:26.158153+0000) 2019-10-18T00:30:26.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:26.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:26.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:26.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:27.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:27.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:27.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:27.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:28.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:28.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:28.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:28.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:29.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:29.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:29.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:29.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:29.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:29.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:30.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:30.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:30.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:30.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:31.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:31.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:31.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:31.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:32.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:32.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:32.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:32.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:32.624 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:30:32.613+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:32.621234+0000) 2019-10-18T00:30:33.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:33.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:33.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:33.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:34.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:30:34.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:34.649790+0000) 2019-10-18T00:30:34.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:34.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:34.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:34.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:34.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:34.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:35.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:35.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:35.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:35.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:36.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:30:36.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:36.075498+0000) 2019-10-18T00:30:36.161 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:30:36.154+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:36.158337+0000) 2019-10-18T00:30:36.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:36.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:36.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:36.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:37.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:37.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:37.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:37.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:38.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:38.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:38.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:38.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:39.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:39.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:39.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:39.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:39.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:39.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:40.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:40.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:40.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:40.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:41.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:41.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:41.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:41.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:42.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:30:42.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:42.621340+0000) 2019-10-18T00:30:42.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:42.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:42.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:42.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:43.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:43.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:43.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:43.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:44.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:30:44.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:44.649972+0000) 2019-10-18T00:30:44.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:44.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:44.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:44.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:44.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:44.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:45.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:45.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:45.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:45.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:46.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:30:46.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:46.075599+0000) 2019-10-18T00:30:46.161 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:30:46.154+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:46.158496+0000) 2019-10-18T00:30:46.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:46.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:46.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:46.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:47.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:47.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:47.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:47.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:48.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:48.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:48.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:48.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:49.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:49.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:49.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:49.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:49.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:49.813+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:50.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:50.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:50.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:50.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:51.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:51.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:51.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:51.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:52.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:30:52.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:52.621455+0000) 2019-10-18T00:30:52.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:52.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:52.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:52.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:53.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:53.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:53.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:53.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:54.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:30:54.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:54.650202+0000) 2019-10-18T00:30:54.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:54.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:54.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:54.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:54.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:54.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:30:55.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:55.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:55.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:55.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:56.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:30:56.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:56.075701+0000) 2019-10-18T00:30:56.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:30:56.154+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:30:56.158679+0000) 2019-10-18T00:30:56.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:56.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:56.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:56.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:57.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:57.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:57.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:57.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:58.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:58.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:58.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:58.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:59.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:59.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:30:59.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:30:59.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:30:59.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:30:59.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:00.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:00.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:00.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:00.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:01.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:01.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:01.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:01.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:02.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:02.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:02.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:02.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:02.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:31:02.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:02.621568+0000) 2019-10-18T00:31:03.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:03.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:03.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:03.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:04.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:04.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:04.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:04.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:04.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:31:04.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:04.650382+0000) 2019-10-18T00:31:04.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:04.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:05.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:05.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:05.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:06.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:31:06.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:06.075805+0000) 2019-10-18T00:31:06.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:31:06.154+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:06.158848+0000) 2019-10-18T00:31:06.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:06.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:06.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:06.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:07.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:07.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:07.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:07.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:08.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:08.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:08.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:08.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:09.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:09.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:09.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:09.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:09.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:09.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:10.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:10.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:10.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:10.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:11.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:11.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:11.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:11.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:12.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:31:12.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:12.621670+0000) 2019-10-18T00:31:12.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:12.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:12.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:12.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:13.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:13.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:13.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:13.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:14.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:31:14.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:14.650563+0000) 2019-10-18T00:31:14.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:14.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:14.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:14.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:14.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:14.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:15.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:15.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:15.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:15.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:16.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:31:16.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:16.075908+0000) 2019-10-18T00:31:16.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:31:16.154+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:16.159041+0000) 2019-10-18T00:31:16.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:16.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:16.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:16.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:17.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:17.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:17.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:17.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:18.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:18.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:18.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:18.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:19.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:19.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:19.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:19.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:19.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:19.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:20.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:20.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:20.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:20.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:21.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:21.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:21.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:21.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:22.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:31:22.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:22.621771+0000) 2019-10-18T00:31:22.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:22.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:22.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:22.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:23.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:23.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:23.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:23.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:24.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:31:24.650+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:24.650742+0000) 2019-10-18T00:31:24.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:24.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:24.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:24.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:24.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:24.816+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:25.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:25.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:25.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:25.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:26.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:31:26.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:26.076010+0000) 2019-10-18T00:31:26.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:31:26.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:26.159248+0000) 2019-10-18T00:31:26.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:26.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:26.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:26.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:27.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:27.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:27.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:27.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:28.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:28.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:28.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:28.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:29.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:29.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:29.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:29.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:29.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:29.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:30.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:30.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:30.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:30.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:31.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:31.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:31.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:31.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:32.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:31:32.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:32.621875+0000) 2019-10-18T00:31:32.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:32.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:32.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:32.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:33.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:33.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:33.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:33.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:34.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:31:34.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:34.650921+0000) 2019-10-18T00:31:34.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:34.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:34.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:34.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:34.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:34.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:35.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:35.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:35.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:35.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:36.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:31:36.071+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:36.076113+0000) 2019-10-18T00:31:36.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:31:36.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:36.159429+0000) 2019-10-18T00:31:36.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:36.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:36.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:36.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:37.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:37.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:37.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:37.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:38.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:38.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:38.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:38.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:39.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:39.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:39.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:39.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:39.826 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:39.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:40.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:40.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:40.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:40.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:41.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:41.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:41.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:41.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:42.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:31:42.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:42.621976+0000) 2019-10-18T00:31:42.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:42.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:42.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:42.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:43.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:43.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:43.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:43.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:44.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:31:44.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:44.651106+0000) 2019-10-18T00:31:44.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:44.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:44.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:44.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:44.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:44.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:45.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:45.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:45.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:45.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:46.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:31:46.071+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:46.076213+0000) 2019-10-18T00:31:46.163 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:31:46.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:46.159614+0000) 2019-10-18T00:31:46.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:46.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:46.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:46.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:47.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:47.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:47.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:47.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:48.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:48.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:48.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:49.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:49.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:49.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:49.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:49.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:49.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:50.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:50.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:50.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:50.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:51.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:51.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:51.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:51.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:52.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:31:52.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:52.622083+0000) 2019-10-18T00:31:52.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:52.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:52.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:52.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:53.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:53.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:53.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:53.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:54.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:31:54.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:54.651294+0000) 2019-10-18T00:31:54.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:54.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:54.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:54.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:54.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:54.823+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:31:55.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:55.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:55.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:55.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:56.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:31:56.071+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:56.076312+0000) 2019-10-18T00:31:56.164 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:31:56.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:31:56.159799+0000) 2019-10-18T00:31:56.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:56.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:56.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:56.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:57.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:57.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:57.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:57.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:58.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:58.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:58.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:58.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:59.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:59.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:31:59.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:31:59.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:31:59.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:31:59.823+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:00.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:00.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:00.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:00.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:01.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:01.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:01.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:01.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:02.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:32:02.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:02.622188+0000) 2019-10-18T00:32:02.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:02.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:02.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:02.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:03.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:03.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:03.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:03.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:04.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:04.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:04.651475+0000) 2019-10-18T00:32:04.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:04.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:04.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:04.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:04.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:04.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:05.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:05.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:05.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:05.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:06.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:32:06.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:06.076412+0000) 2019-10-18T00:32:06.163 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:32:06.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:06.160006+0000) 2019-10-18T00:32:06.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:06.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:06.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:06.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:07.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:07.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:07.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:07.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:08.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:08.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:08.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:08.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:09.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:09.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:09.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:09.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:09.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:09.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:10.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:10.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:10.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:10.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:11.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:11.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:11.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:11.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:12.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:32:12.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:12.622288+0000) 2019-10-18T00:32:12.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:12.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:12.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:12.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:13.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:13.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:13.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:14.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:14.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:14.651677+0000) 2019-10-18T00:32:14.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:14.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:14.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:14.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:14.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:14.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:15.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:15.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:15.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:15.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:16.079 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:32:16.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:16.076511+0000) 2019-10-18T00:32:16.163 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:32:16.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:16.160217+0000) 2019-10-18T00:32:16.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:16.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:16.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:16.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:17.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:17.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:17.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:17.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:18.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:18.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:18.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:18.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:19.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:19.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:19.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:19.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:19.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:19.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:20.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:20.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:20.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:20.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:21.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:21.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:21.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:21.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:22.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:32:22.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:22.622392+0000) 2019-10-18T00:32:22.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:22.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:22.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:22.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:23.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:23.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:23.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:23.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:24.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:24.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:24.651877+0000) 2019-10-18T00:32:24.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:24.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:21:53.934803+0000 (oldest deadline 2019-10-18T00:22:13.934803+0000) 2019-10-18T00:32:24.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:24.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:24.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:24.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:25.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:25.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:26.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:32:26.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:26.076608+0000) 2019-10-18T00:32:26.163 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:32:26.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:26.160407+0000) 2019-10-18T00:32:26.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:26.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:27.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:27.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:28.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:28.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:29.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:29.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:29.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:29.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:30.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:30.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:31.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:31.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:32.625 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:32:32.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:32.622493+0000) 2019-10-18T00:32:32.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:32.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:33.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:33.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:34.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:34.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:34.652076+0000) 2019-10-18T00:32:34.656 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:34.649+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:34.653110+0000) 2019-10-18T00:32:34.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:34.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:34.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:34.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:35.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:35.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:36.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:32:36.070+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:36.076712+0000) 2019-10-18T00:32:36.164 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:32:36.157+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:36.160617+0000) 2019-10-18T00:32:36.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:36.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:37.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:37.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:38.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:38.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:39.156 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:39.149+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:39.152484+0000) 2019-10-18T00:32:39.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:39.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:39.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:39.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:40.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:40.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:41.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:41.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:42.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:32:42.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:42.622595+0000) 2019-10-18T00:32:42.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:42.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:43.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:43.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:44.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:44.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:44.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:44.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:45.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:45.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:46.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:32:46.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:46.076811+0000) 2019-10-18T00:32:46.164 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:32:46.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:46.160827+0000) 2019-10-18T00:32:46.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:46.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:47.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:47.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:48.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:48.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:49.156 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:49.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:49.152668+0000) 2019-10-18T00:32:49.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:49.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:49.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:49.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:49.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:49.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:50.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:50.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:50.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:50.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:51.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:51.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:51.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:51.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:52.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:32:52.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:52.622702+0000) 2019-10-18T00:32:52.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:52.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:52.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:52.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:53.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:53.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:53.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:53.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:54.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:54.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:54.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:54.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:54.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:54.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:32:55.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:55.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:55.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:55.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:56.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:32:56.069+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:56.076914+0000) 2019-10-18T00:32:56.164 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:32:56.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:56.161043+0000) 2019-10-18T00:32:56.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:56.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:56.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:56.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:57.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:57.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:57.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:57.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:58.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:58.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:58.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:58.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:59.156 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:32:59.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:32:59.152851+0000) 2019-10-18T00:32:59.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:59.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:32:59.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:32:59.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:32:59.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:32:59.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:00.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:00.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:00.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:00.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:01.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:01.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:01.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:01.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:02.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:02.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:02.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:02.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:02.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:33:02.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:02.622799+0000) 2019-10-18T00:33:03.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:03.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:03.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:03.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:04.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:04.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:04.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:04.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:04.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:04.837+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:05.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:05.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:05.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:05.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:06.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:33:06.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:06.077025+0000) 2019-10-18T00:33:06.164 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:33:06.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:06.161251+0000) 2019-10-18T00:33:06.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:06.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:06.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:06.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:07.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:07.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:07.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:07.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:08.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:08.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:08.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:08.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:09.158 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:33:09.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:09.153038+0000) 2019-10-18T00:33:09.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:09.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:09.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:09.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:09.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:09.837+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:10.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:10.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:10.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:10.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:11.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:11.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:11.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:11.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:12.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:12.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:12.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:12.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:12.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:33:12.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:12.622900+0000) 2019-10-18T00:33:13.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:13.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:13.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:13.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:14.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:14.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:14.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:14.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:14.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:14.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:15.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:15.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:15.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:15.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:16.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:33:16.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:16.077136+0000) 2019-10-18T00:33:16.164 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:33:16.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:16.161439+0000) 2019-10-18T00:33:16.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:16.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:16.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:16.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:17.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:17.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:17.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:17.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:18.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:18.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:18.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:18.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:19.156 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:33:19.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:19.153220+0000) 2019-10-18T00:33:19.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:19.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:19.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:19.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:19.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:19.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:20.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:20.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:20.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:20.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:21.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:21.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:21.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:21.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:22.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:22.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:22.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:22.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:22.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:33:22.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:22.623004+0000) 2019-10-18T00:33:23.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:23.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:23.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:23.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:24.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:24.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:24.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:24.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:24.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:24.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:25.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:25.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:25.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:25.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:26.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:33:26.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:26.077237+0000) 2019-10-18T00:33:26.165 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:33:26.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:26.161644+0000) 2019-10-18T00:33:26.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:26.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:26.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:26.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:27.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:27.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:27.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:27.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:28.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:28.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:28.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:28.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:29.156 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:33:29.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:29.153400+0000) 2019-10-18T00:33:29.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:29.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:29.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:29.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:29.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:29.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:30.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:30.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:30.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:30.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:31.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:31.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:31.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:31.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:32.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:33:32.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:32.623100+0000) 2019-10-18T00:33:32.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:32.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:32.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:32.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:33.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:33.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:33.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:33.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:34.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:34.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:34.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:34.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:34.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:34.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:35.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:35.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:35.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:35.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:36.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:33:36.072+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:36.077340+0000) 2019-10-18T00:33:36.165 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:33:36.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:36.161867+0000) 2019-10-18T00:33:36.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:36.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:36.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:36.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:37.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:37.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:37.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:37.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:38.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:38.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:38.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:38.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:39.156 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:33:39.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:39.153610+0000) 2019-10-18T00:33:39.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:39.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:39.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:39.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:39.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:39.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:40.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:40.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:40.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:40.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:41.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:41.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:41.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:41.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:42.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:42.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:42.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:42.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:33:42.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:42.623195+0000) 2019-10-18T00:33:43.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:43.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:43.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:43.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:44.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:44.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:44.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:44.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:44.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:44.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:45.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:45.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:45.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:45.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:46.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:33:46.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:46.077450+0000) 2019-10-18T00:33:46.165 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:33:46.156+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:46.162053+0000) 2019-10-18T00:33:46.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:46.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:46.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:46.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:47.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:47.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:47.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:47.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:48.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:48.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:48.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:48.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:49.157 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:33:49.148+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:49.153819+0000) 2019-10-18T00:33:49.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:49.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:49.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:49.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:49.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:49.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:50.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:50.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:50.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:50.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:51.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:51.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:51.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:51.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:52.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:33:52.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:52.623294+0000) 2019-10-18T00:33:52.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:52.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:52.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:52.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:53.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:53.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:53.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:53.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:54.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:54.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:54.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:54.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:54.850 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:54.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:33:55.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:55.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:55.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:55.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:56.080 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:33:56.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:56.077550+0000) 2019-10-18T00:33:56.165 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:33:56.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:56.162259+0000) 2019-10-18T00:33:56.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:56.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:56.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:56.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:57.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:57.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:57.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:57.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:58.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:58.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:58.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:58.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:59.157 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:33:59.147+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:33:59.154001+0000) 2019-10-18T00:33:59.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:59.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:33:59.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:33:59.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:33:59.850 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:33:59.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:00.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:00.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:00.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:00.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:01.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:01.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:01.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:01.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:02.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:34:02.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:02.623390+0000) 2019-10-18T00:34:02.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:02.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:02.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:02.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:03.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:03.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:03.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:03.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:04.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:04.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:04.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:04.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:04.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:04.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:05.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:05.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:05.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:05.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:06.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:34:06.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:06.077648+0000) 2019-10-18T00:34:06.165 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:34:06.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:06.162412+0000) 2019-10-18T00:34:06.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:06.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:06.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:06.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:07.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:07.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:07.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:07.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:08.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:08.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:08.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:08.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:09.159 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:34:09.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:09.154228+0000) 2019-10-18T00:34:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:09.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:09.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:09.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:09.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:10.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:10.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:10.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:10.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:11.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:11.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:11.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:11.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:12.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:12.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:12.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:12.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:12.626 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:34:12.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:12.623491+0000) 2019-10-18T00:34:13.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:13.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:13.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:13.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:14.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:14.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:14.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:14.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:14.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:14.847+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:15.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:15.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:15.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:15.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:16.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:34:16.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:16.077761+0000) 2019-10-18T00:34:16.165 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:34:16.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:16.162583+0000) 2019-10-18T00:34:16.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:16.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:16.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:16.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:17.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:17.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:17.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:17.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:18.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:18.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:18.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:18.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:19.157 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:34:19.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:19.154424+0000) 2019-10-18T00:34:19.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:19.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:19.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:19.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:19.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:19.847+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:20.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:20.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:20.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:20.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:21.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:21.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:21.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:21.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:22.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:22.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:22.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:22.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:22.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:34:22.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:22.623598+0000) 2019-10-18T00:34:23.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:23.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:23.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:23.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:24.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:24.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:24.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:24.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:24.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:24.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:25.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:25.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:25.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:25.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:26.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:34:26.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:26.077860+0000) 2019-10-18T00:34:26.166 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:34:26.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:26.162750+0000) 2019-10-18T00:34:26.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:26.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:26.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:26.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:27.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:27.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:27.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:27.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:28.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:28.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:28.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:28.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:29.158 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:34:29.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:29.154617+0000) 2019-10-18T00:34:29.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:29.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:29.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:29.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:29.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:29.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:30.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:30.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:31.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:31.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:31.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:31.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:32.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:32.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:32.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:32.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:32.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:34:32.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:32.623702+0000) 2019-10-18T00:34:33.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:33.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:33.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:33.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:34.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:34.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:34.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:34.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:34.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:34.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:35.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:35.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:35.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:35.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:36.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:34:36.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:36.077959+0000) 2019-10-18T00:34:36.166 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:34:36.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:36.162932+0000) 2019-10-18T00:34:36.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:36.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:36.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:36.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:37.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:37.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:37.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:37.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:38.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:38.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:38.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:38.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:39.158 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:34:39.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:39.154796+0000) 2019-10-18T00:34:39.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:39.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:39.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:39.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:39.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:39.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:40.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:40.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:40.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:40.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:41.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:41.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:41.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:41.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:42.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:42.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:42.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:42.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:42.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:34:42.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:42.623807+0000) 2019-10-18T00:34:43.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:43.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:43.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:43.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:44.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:44.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:44.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:44.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:44.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:44.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:45.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:45.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:45.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:45.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:46.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:34:46.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:46.078057+0000) 2019-10-18T00:34:46.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:34:46.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:46.163115+0000) 2019-10-18T00:34:46.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:46.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:46.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:46.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:47.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:47.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:47.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:47.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:48.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:48.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:48.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:48.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:49.158 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:34:49.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:49.154975+0000) 2019-10-18T00:34:49.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:49.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:49.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:49.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:49.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:49.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:50.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:50.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:50.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:50.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:51.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:51.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:51.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:51.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:52.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:52.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:52.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:52.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:34:52.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:52.623918+0000) 2019-10-18T00:34:53.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:53.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:53.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:53.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:54.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:54.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:54.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:54.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:54.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:54.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:34:55.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:55.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:55.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:55.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:56.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:34:56.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:56.078159+0000) 2019-10-18T00:34:56.166 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:34:56.159+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:56.163305+0000) 2019-10-18T00:34:56.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:56.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:56.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:56.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:57.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:57.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:57.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:57.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:58.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:58.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:58.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:58.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:59.158 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:34:59.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:34:59.155180+0000) 2019-10-18T00:34:59.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:59.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:34:59.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:34:59.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:34:59.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:34:59.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:00.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:00.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:00.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:00.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:01.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:01.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:01.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:01.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:02.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:02.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:02.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:02.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:02.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:35:02.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:02.624022+0000) 2019-10-18T00:35:03.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:03.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:03.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:03.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:04.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:04.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:04.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:04.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:05.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:05.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:05.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:05.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:06.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:35:06.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:06.078255+0000) 2019-10-18T00:35:06.166 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:35:06.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:06.163519+0000) 2019-10-18T00:35:06.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:06.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:06.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:06.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:07.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:07.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:07.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:07.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:08.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:08.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:08.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:08.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:09.158 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:35:09.150+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:09.155357+0000) 2019-10-18T00:35:09.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:09.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:09.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:09.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:09.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:09.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:10.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:10.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:10.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:10.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:11.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:11.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:11.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:11.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:12.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:35:12.617+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:12.624123+0000) 2019-10-18T00:35:12.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:12.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:12.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:12.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:13.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:13.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:13.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:13.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:14.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:14.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:14.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:14.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:14.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:14.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:15.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:15.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:15.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:15.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:16.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:35:16.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:16.078356+0000) 2019-10-18T00:35:16.166 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:35:16.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:16.163733+0000) 2019-10-18T00:35:16.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:16.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:16.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:16.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:17.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:17.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:17.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:17.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:18.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:18.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:18.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:18.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:19.159 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:35:19.150+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:19.155539+0000) 2019-10-18T00:35:19.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:19.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:19.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:19.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:19.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:19.861+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:20.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:20.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:20.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:20.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:21.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:21.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:21.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:21.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:22.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:35:22.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:22.624228+0000) 2019-10-18T00:35:22.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:22.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:22.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:22.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:23.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:23.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:23.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:23.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:24.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:24.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:24.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:24.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:24.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:24.861+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:25.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:25.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:25.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:25.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:26.081 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:35:26.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:26.078457+0000) 2019-10-18T00:35:26.167 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:35:26.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:26.163948+0000) 2019-10-18T00:35:26.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:26.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:26.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:26.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:27.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:27.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:27.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:27.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:28.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:28.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:28.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:28.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:29.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:35:29.150+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:29.155746+0000) 2019-10-18T00:35:29.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:29.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:29.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:29.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:29.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:29.861+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:30.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:30.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:30.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:30.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:31.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:31.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:31.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:31.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:32.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:32.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:32.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:32.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:32.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:35:32.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:32.624337+0000) 2019-10-18T00:35:33.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:33.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:33.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:33.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:34.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:34.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:34.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:34.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:34.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:34.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:35.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:35.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:35.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:35.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:36.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:35:36.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:36.078563+0000) 2019-10-18T00:35:36.167 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:35:36.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:36.164162+0000) 2019-10-18T00:35:36.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:36.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:36.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:36.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:37.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:37.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:37.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:37.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:38.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:38.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:38.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:38.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:39.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:35:39.150+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:39.155938+0000) 2019-10-18T00:35:39.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:39.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:39.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:39.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:39.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:39.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:40.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:40.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:40.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:40.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:41.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:41.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:41.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:41.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:42.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:42.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:42.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:42.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:42.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:35:42.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:42.624439+0000) 2019-10-18T00:35:43.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:43.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:43.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:43.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:44.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:44.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:44.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:44.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:44.868 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:44.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:45.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:45.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:45.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:45.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:46.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:35:46.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:46.078661+0000) 2019-10-18T00:35:46.167 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:35:46.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:46.164337+0000) 2019-10-18T00:35:46.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:46.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:46.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:46.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:47.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:47.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:47.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:47.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:48.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:48.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:48.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:48.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:49.159 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:35:49.150+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:49.156149+0000) 2019-10-18T00:35:49.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:49.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:49.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:49.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:49.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:49.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:50.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:50.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:50.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:50.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:51.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:51.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:51.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:51.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:52.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:52.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:52.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:52.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:52.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:35:52.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:52.624539+0000) 2019-10-18T00:35:53.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:53.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:53.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:53.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:54.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:54.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:54.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:54.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:54.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:54.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:35:55.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:55.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:55.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:55.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:56.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:35:56.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:56.078762+0000) 2019-10-18T00:35:56.167 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:35:56.158+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:56.164544+0000) 2019-10-18T00:35:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:56.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:56.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:57.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:57.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:57.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:57.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:58.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:58.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:58.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:58.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:59.159 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:35:59.150+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:35:59.156357+0000) 2019-10-18T00:35:59.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:59.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:35:59.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:35:59.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:35:59.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:35:59.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:00.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:00.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:00.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:00.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:01.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:01.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:01.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:01.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:02.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:02.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:02.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:02.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:02.627 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:36:02.620+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:02.624639+0000) 2019-10-18T00:36:03.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:03.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:03.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:03.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:04.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:04.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:04.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:04.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:04.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:04.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:05.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:05.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:05.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:06.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:36:06.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:06.078862+0000) 2019-10-18T00:36:06.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:36:06.162+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:06.164725+0000) 2019-10-18T00:36:06.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:06.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:06.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:06.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:07.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:07.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:07.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:07.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:08.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:08.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:08.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:08.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:09.159 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:36:09.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:09.156537+0000) 2019-10-18T00:36:09.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:09.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:09.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:09.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:09.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:09.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:10.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:10.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:10.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:10.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:11.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:11.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:11.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:11.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:12.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:12.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:12.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:12.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:12.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:36:12.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:12.624750+0000) 2019-10-18T00:36:13.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:13.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:13.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:13.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:14.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:14.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:14.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:14.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:14.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:14.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:15.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:15.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:15.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:15.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:16.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:36:16.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:16.078960+0000) 2019-10-18T00:36:16.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:36:16.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:16.164942+0000) 2019-10-18T00:36:16.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:16.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:16.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:16.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:17.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:17.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:18.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:18.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:18.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:18.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:19.159 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:36:19.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:19.156718+0000) 2019-10-18T00:36:19.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:19.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:19.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:19.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:19.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:19.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:20.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:20.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:20.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:21.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:21.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:21.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:21.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:22.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:22.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:22.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:22.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:22.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:36:22.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:22.624858+0000) 2019-10-18T00:36:23.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:23.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:23.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:23.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:24.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:24.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:24.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:24.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:24.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:24.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:25.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:25.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:25.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:25.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:26.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:36:26.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:26.079061+0000) 2019-10-18T00:36:26.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:36:26.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:26.165153+0000) 2019-10-18T00:36:26.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:26.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:26.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:26.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:27.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:27.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:27.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:27.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:28.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:28.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:28.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:28.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:29.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:36:29.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:29.156909+0000) 2019-10-18T00:36:29.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:29.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:29.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:29.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:29.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:29.871+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:30.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:30.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:30.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:30.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:31.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:31.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:31.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:31.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:32.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:32.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:32.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:32.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:32.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:36:32.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:32.624964+0000) 2019-10-18T00:36:33.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:33.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:33.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:33.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:34.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:34.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:34.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:34.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:34.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:34.871+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:35.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:35.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:35.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:35.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:36.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:36:36.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:36.079165+0000) 2019-10-18T00:36:36.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:36:36.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:36.165375+0000) 2019-10-18T00:36:36.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:36.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:36.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:36.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:37.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:37.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:37.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:37.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:38.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:38.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:38.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:38.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:39.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:36:39.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:39.157092+0000) 2019-10-18T00:36:39.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:39.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:39.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:39.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:39.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:39.871+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:40.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:40.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:40.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:40.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:41.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:41.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:41.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:41.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:42.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:42.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:42.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:42.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:42.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:36:42.619+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:42.625062+0000) 2019-10-18T00:36:43.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:43.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:43.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:43.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:44.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:44.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:44.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:44.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:44.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:44.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:45.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:45.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:45.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:45.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:46.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:36:46.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:46.079264+0000) 2019-10-18T00:36:46.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:36:46.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:46.165550+0000) 2019-10-18T00:36:46.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:46.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:46.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:46.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:47.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:47.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:47.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:47.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:48.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:48.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:48.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:48.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:49.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:36:49.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:49.157277+0000) 2019-10-18T00:36:49.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:49.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:49.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:49.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:49.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:49.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:50.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:50.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:50.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:50.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:51.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:51.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:51.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:51.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:52.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:52.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:52.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:52.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:52.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:36:52.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:52.625158+0000) 2019-10-18T00:36:53.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:53.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:53.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:53.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:54.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:54.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:54.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:54.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:36:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:55.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:55.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:56.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:36:56.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:56.079365+0000) 2019-10-18T00:36:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:56.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:56.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:56.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:56.168 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:36:56.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:56.165762+0000) 2019-10-18T00:36:57.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:57.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:57.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:57.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:58.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:58.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:58.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:58.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:59.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:36:59.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:36:59.157489+0000) 2019-10-18T00:36:59.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:59.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:36:59.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:36:59.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:36:59.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:36:59.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:00.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:00.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:01.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:01.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:01.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:01.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:02.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:02.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:02.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:02.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:02.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:37:02.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:02.625259+0000) 2019-10-18T00:37:03.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:03.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:03.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:03.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:04.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:04.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:04.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:04.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:04.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:04.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:05.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:05.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:05.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:05.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:06.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:06.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:06.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:06.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:06.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:37:06.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:06.079461+0000) 2019-10-18T00:37:06.169 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:37:06.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:06.165953+0000) 2019-10-18T00:37:07.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:07.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:07.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:07.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:08.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:07.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:08.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:07.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:09.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:09.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:09.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:09.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:09.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:37:09.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:09.157668+0000) 2019-10-18T00:37:09.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:09.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:10.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:10.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:10.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:10.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:11.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:11.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:11.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:11.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:12.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:12.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:12.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:12.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:12.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:37:12.618+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:12.625360+0000) 2019-10-18T00:37:13.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:13.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:13.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:13.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:14.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:14.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:14.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:14.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:14.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:14.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:15.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:15.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:15.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:15.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:16.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:16.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:16.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:16.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:16.082 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:37:16.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:16.079565+0000) 2019-10-18T00:37:16.169 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:37:16.161+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:16.166140+0000) 2019-10-18T00:37:17.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:17.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:17.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:17.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:18.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:18.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:18.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:18.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:19.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:19.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:19.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:19.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:19.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:37:19.153+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:19.157859+0000) 2019-10-18T00:37:19.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:19.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:19.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:19.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:19.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:19.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:21.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:20.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:21.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:20.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:21.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:21.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:22.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:37:22.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:22.625451+0000) 2019-10-18T00:37:22.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:22.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:22.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:22.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:23.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:23.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:23.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:23.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:24.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:24.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:24.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:24.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:24.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:24.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:25.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:25.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:25.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:25.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:26.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:37:26.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:26.079667+0000) 2019-10-18T00:37:26.169 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:37:26.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:26.166319+0000) 2019-10-18T00:37:26.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:26.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:26.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:26.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:27.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:27.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:27.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:27.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:28.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:28.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:28.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:28.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:29.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:37:29.152+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:29.158070+0000) 2019-10-18T00:37:29.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:29.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:29.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:29.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:29.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:29.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:30.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:30.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:30.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:30.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:31.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:31.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:31.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:31.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:32.628 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:37:32.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:32.625558+0000) 2019-10-18T00:37:32.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:32.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:32.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:32.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:33.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:33.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:33.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:33.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:34.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:34.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:34.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:34.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:34.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:34.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:35.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:35.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:35.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:35.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:36.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:37:36.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:36.079779+0000) 2019-10-18T00:37:36.169 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:37:36.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:36.166475+0000) 2019-10-18T00:37:36.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:36.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:36.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:36.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:37.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:37.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:37.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:37.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:38.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:38.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:38.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:38.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:39.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:37:39.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:39.158271+0000) 2019-10-18T00:37:39.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:39.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:39.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:39.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:39.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:39.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:40.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:40.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:40.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:40.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:41.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:41.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:41.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:41.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:42.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:37:42.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:42.625657+0000) 2019-10-18T00:37:42.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:42.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:42.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:42.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:43.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:43.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:43.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:43.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:44.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:44.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:44.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:44.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:44.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:44.885+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:45.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:45.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:45.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:45.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:46.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:37:46.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:46.079884+0000) 2019-10-18T00:37:46.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:37:46.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:46.166646+0000) 2019-10-18T00:37:46.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:46.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:46.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:46.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:47.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:47.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:47.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:47.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:48.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:48.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:48.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:48.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:49.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:37:49.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:49.158455+0000) 2019-10-18T00:37:49.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:49.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:49.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:49.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:49.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:49.885+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:50.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:50.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:50.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:50.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:51.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:51.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:51.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:51.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:52.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:37:52.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:52.625761+0000) 2019-10-18T00:37:52.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:52.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:52.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:52.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:53.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:53.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:53.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:53.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:54.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:54.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:54.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:54.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:54.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:54.885+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:37:55.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:55.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:55.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:55.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:56.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:37:56.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:56.079985+0000) 2019-10-18T00:37:56.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:37:56.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:56.166853+0000) 2019-10-18T00:37:56.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:56.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:56.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:56.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:57.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:57.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:57.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:57.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:58.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:58.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:58.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:58.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:59.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:37:59.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:37:59.158639+0000) 2019-10-18T00:37:59.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:59.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:37:59.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:37:59.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:37:59.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:37:59.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:00.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:00.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:00.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:00.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:01.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:01.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:01.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:01.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:02.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:38:02.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:02.625862+0000) 2019-10-18T00:38:02.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:02.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:02.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:02.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:03.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:03.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:03.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:03.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:04.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:04.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:04.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:04.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:04.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:04.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:05.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:05.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:05.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:05.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:06.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:38:06.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:06.080084+0000) 2019-10-18T00:38:06.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:38:06.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:06.167035+0000) 2019-10-18T00:38:06.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:06.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:06.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:06.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:07.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:07.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:07.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:07.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:08.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:08.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:08.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:08.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:09.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:38:09.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:09.158820+0000) 2019-10-18T00:38:09.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:09.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:09.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:09.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:09.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:09.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:10.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:10.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:10.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:10.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:11.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:11.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:11.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:11.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:12.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:38:12.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:12.625963+0000) 2019-10-18T00:38:12.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:12.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:12.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:12.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:13.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:13.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:13.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:13.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:14.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:14.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:14.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:14.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:14.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:14.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:15.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:15.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:15.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:15.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:16.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:38:16.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:16.080191+0000) 2019-10-18T00:38:16.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:38:16.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:16.167219+0000) 2019-10-18T00:38:16.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:16.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:16.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:16.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:17.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:17.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:17.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:17.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:18.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:18.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:18.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:18.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:19.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:38:19.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:19.159033+0000) 2019-10-18T00:38:19.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:19.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:19.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:19.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:19.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:19.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:20.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:20.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:20.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:20.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:21.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:21.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:21.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:22.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:22.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:22.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:22.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:22.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:38:22.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:22.626066+0000) 2019-10-18T00:38:23.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:23.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:23.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:23.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:24.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:24.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:24.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:24.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:24.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:24.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:25.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:25.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:25.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:25.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:26.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:38:26.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:26.080290+0000) 2019-10-18T00:38:26.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:38:26.164+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:26.167406+0000) 2019-10-18T00:38:26.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:26.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:26.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:26.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:27.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:27.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:27.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:27.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:28.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:28.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:28.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:28.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:29.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:38:29.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:29.159241+0000) 2019-10-18T00:38:29.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:29.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:29.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:29.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:29.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:29.892+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:30.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:30.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:30.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:30.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:31.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:31.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:31.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:31.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:32.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:38:32.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:32.626172+0000) 2019-10-18T00:38:32.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:32.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:32.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:32.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:33.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:33.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:33.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:34.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:34.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:34.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:34.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:34.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:34.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:35.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:35.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:35.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:35.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:36.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:38:36.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:36.080393+0000) 2019-10-18T00:38:36.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:38:36.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:36.167597+0000) 2019-10-18T00:38:36.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:36.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:36.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:36.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:37.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:37.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:37.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:37.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:38.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:38.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:38.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:38.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:39.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:38:39.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:39.159419+0000) 2019-10-18T00:38:39.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:39.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:39.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:39.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:39.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:39.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:40.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:40.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:40.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:40.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:41.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:41.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:41.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:41.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:42.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:38:42.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:42.626277+0000) 2019-10-18T00:38:42.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:42.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:42.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:42.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:43.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:43.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:43.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:43.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:44.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:44.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:44.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:44.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:44.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:44.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:45.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:45.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:45.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:45.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:46.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:38:46.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:46.080494+0000) 2019-10-18T00:38:46.170 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:38:46.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:46.167801+0000) 2019-10-18T00:38:46.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:46.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:46.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:46.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:47.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:47.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:47.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:47.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:48.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:48.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:48.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:48.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:49.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:38:49.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:49.159623+0000) 2019-10-18T00:38:49.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:49.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:49.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:49.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:49.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:49.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:50.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:50.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:50.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:50.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:51.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:51.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:51.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:51.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:52.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:38:52.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:52.626386+0000) 2019-10-18T00:38:52.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:52.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:52.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:52.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:53.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:53.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:53.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:53.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:54.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:54.895+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:38:54.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:54.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:54.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:54.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:55.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:55.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:55.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:55.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:56.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:38:56.075+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:56.080604+0000) 2019-10-18T00:38:56.171 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:38:56.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:56.168000+0000) 2019-10-18T00:38:56.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:56.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:56.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:56.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:57.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:57.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:57.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:57.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:58.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:58.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:58.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:58.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:59.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:38:59.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:38:59.159804+0000) 2019-10-18T00:38:59.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:59.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:38:59.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:38:59.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:38:59.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:38:59.895+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:00.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:00.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:00.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:00.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:01.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:01.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:01.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:01.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:02.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:39:02.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:02.626495+0000) 2019-10-18T00:39:02.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:02.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:02.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:02.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:03.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:03.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:03.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:03.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:04.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:04.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:04.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:04.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:04.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:04.895+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:05.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:05.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:05.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:05.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:06.083 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:39:06.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:06.080706+0000) 2019-10-18T00:39:06.171 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:06.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:06.168212+0000) 2019-10-18T00:39:06.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:06.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:06.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:06.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:07.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:07.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:07.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:07.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:08.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:08.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:08.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:08.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:09.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:39:09.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:09.159982+0000) 2019-10-18T00:39:09.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:09.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:09.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:09.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:09.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:09.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:10.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:10.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:10.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:10.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:11.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:11.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:11.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:11.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:12.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:39:12.622+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:12.626600+0000) 2019-10-18T00:39:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:12.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:12.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:13.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:13.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:13.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:13.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:14.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:14.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:14.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:14.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:14.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:14.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:15.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:15.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:15.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:15.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:16.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:39:16.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:16.080804+0000) 2019-10-18T00:39:16.171 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:16.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:16.168408+0000) 2019-10-18T00:39:16.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:16.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:16.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:16.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:17.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:17.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:17.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:17.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:18.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:18.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:18.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:18.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:19.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:39:19.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:19.160164+0000) 2019-10-18T00:39:19.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:19.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:19.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:19.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:19.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:19.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:20.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:20.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:20.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:20.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:21.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:21.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:21.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:21.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:22.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:22.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:22.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:22.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:22.629 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:39:22.622+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:22.626709+0000) 2019-10-18T00:39:23.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:23.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:23.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:23.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:24.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:24.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:24.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:24.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:24.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:24.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:25.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:25.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:25.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:25.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:26.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:39:26.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:26.080902+0000) 2019-10-18T00:39:26.171 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:26.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:26.168593+0000) 2019-10-18T00:39:26.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:26.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:26.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:26.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:27.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:27.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:27.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:27.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:28.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:28.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:28.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:28.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:29.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:39:29.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:29.160348+0000) 2019-10-18T00:39:29.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:29.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:29.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:29.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:29.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:29.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:30.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:30.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:30.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:30.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:31.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:31.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:31.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:31.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:32.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:39:32.622+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:32.626816+0000) 2019-10-18T00:39:32.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:32.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:32.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:32.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:33.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:33.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:33.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:33.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:34.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:34.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:34.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:34.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:34.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:34.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:35.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:35.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:35.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:35.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:36.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:39:36.074+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:36.080998+0000) 2019-10-18T00:39:36.171 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:36.163+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:36.168770+0000) 2019-10-18T00:39:36.172 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:36.163+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:36.169861+0000) 2019-10-18T00:39:36.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:36.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:36.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:36.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:37.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:37.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:37.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:37.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:38.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:38.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:38.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:38.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:39.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:39:39.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:39.160515+0000) 2019-10-18T00:39:39.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:39.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:39.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:39.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:39.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:39.902+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:40.672 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:40.663+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:40.669215+0000) 2019-10-18T00:39:40.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:40.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:40.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:40.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:41.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:41.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:41.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:41.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:42.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:42.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:42.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:42.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:42.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:39:42.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:42.626916+0000) 2019-10-18T00:39:43.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:43.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:43.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:43.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:44.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:44.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:44.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:44.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:44.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:44.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:45.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:45.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:45.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:45.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:46.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:39:46.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:46.081094+0000) 2019-10-18T00:39:46.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:46.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:46.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:46.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:47.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:47.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:47.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:47.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:48.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:48.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:48.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:48.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:49.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:39:49.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:49.160701+0000) 2019-10-18T00:39:49.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:49.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:49.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:49.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:49.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:49.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:50.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:50.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:50.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:50.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:50.672 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:39:50.662+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:50.669420+0000) 2019-10-18T00:39:51.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:51.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:51.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:51.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:52.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:52.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:52.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:52.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:52.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:39:52.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:52.627016+0000) 2019-10-18T00:39:53.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:53.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:53.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:53.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:54.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:54.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:54.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:54.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:54.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:54.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:39:55.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:55.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:55.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:55.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:56.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:39:56.073+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:56.081190+0000) 2019-10-18T00:39:56.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:56.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:56.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:56.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:57.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:57.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:57.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:57.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:58.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:58.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:58.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:58.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:59.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:39:59.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:39:59.160883+0000) 2019-10-18T00:39:59.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:59.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:39:59.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:39:59.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:39:59.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:39:59.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:00.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:00.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:00.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:00.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:00.672 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:40:00.662+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:00.669605+0000) 2019-10-18T00:40:01.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:01.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:01.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:01.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:02.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:02.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:02.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:02.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:02.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:40:02.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:02.627120+0000) 2019-10-18T00:40:03.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:03.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:03.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:03.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:04.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:04.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:04.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:04.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:04.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:04.909+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:05.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:05.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:05.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:05.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:06.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:40:06.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:06.081289+0000) 2019-10-18T00:40:06.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:06.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:06.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:06.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:07.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:07.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:07.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:07.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:08.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:08.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:08.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:08.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:09.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:40:09.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:09.161070+0000) 2019-10-18T00:40:09.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:09.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:09.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:09.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:09.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:09.909+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:10.672 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:40:10.662+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:10.669818+0000) 2019-10-18T00:40:10.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:10.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:10.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:10.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:11.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:11.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:11.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:11.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:12.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:40:12.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:12.627223+0000) 2019-10-18T00:40:12.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:12.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:12.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:12.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:13.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:13.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:13.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:13.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:14.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:14.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:14.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:14.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:14.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:14.909+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:15.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:15.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:15.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:15.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:16.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:40:16.076+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:16.081390+0000) 2019-10-18T00:40:16.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:16.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:16.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:16.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:17.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:17.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:17.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:17.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:18.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:18.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:18.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:18.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:19.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:40:19.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:19.161215+0000) 2019-10-18T00:40:19.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:19.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:19.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:19.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:19.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:19.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:20.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:20.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:20.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:20.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:20.672 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:40:20.662+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:20.670002+0000) 2019-10-18T00:40:21.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:21.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:21.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:21.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:22.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:22.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:22.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:22.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:22.630 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:40:22.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:22.627324+0000) 2019-10-18T00:40:23.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:23.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:23.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:23.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:24.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:24.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:24.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:24.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:24.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:24.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:25.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:25.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:25.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:25.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:26.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:40:26.080+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:26.081482+0000) 2019-10-18T00:40:26.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:26.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:26.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:26.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:27.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:27.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:27.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:27.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:28.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:28.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:28.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:28.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:29.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:40:29.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:29.161406+0000) 2019-10-18T00:40:29.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:29.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:29.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:29.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:29.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:29.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:30.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:30.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:30.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:30.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:30.673 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:40:30.666+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:30.670226+0000) 2019-10-18T00:40:31.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:31.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:31.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:31.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:32.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:32.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:32.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:32.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:32.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:40:32.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:32.627486+0000) 2019-10-18T00:40:33.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:33.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:33.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:33.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:34.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:34.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:34.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:34.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:34.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:34.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:35.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:35.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:35.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:35.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:36.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:40:36.080+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:36.081578+0000) 2019-10-18T00:40:36.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:36.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:36.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:36.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:37.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:37.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:37.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:37.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:38.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:38.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:38.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:38.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:39.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:40:39.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:39.161595+0000) 2019-10-18T00:40:39.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:39.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:39.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:39.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:39.919 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:39.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:40.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:40.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:40.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:40.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:40.673 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:40:40.666+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:40.670404+0000) 2019-10-18T00:40:41.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:41.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:41.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:41.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:42.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:42.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:42.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:42.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:42.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:40:42.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:42.627699+0000) 2019-10-18T00:40:43.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:43.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:43.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:43.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:44.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:44.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:44.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:44.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:44.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:44.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:45.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:45.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:45.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:45.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:46.084 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:40:46.080+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:46.081680+0000) 2019-10-18T00:40:46.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:46.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:46.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:46.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:47.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:47.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:47.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:47.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:48.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:48.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:48.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:48.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:49.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:40:49.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:49.161748+0000) 2019-10-18T00:40:49.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:49.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:49.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:49.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:49.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:49.916+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:50.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:50.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:50.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:50.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:50.673 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:40:50.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:50.670553+0000) 2019-10-18T00:40:51.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:51.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:51.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:51.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:52.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:40:52.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:52.627807+0000) 2019-10-18T00:40:52.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:52.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:52.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:52.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:53.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:53.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:53.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:53.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:54.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:54.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:54.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:54.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:54.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:54.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:40:55.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:55.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:55.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:55.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:56.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:40:56.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:56.081783+0000) 2019-10-18T00:40:56.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:56.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:56.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:56.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:57.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:57.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:57.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:57.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:58.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:58.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:58.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:58.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:59.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:40:59.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:40:59.161928+0000) 2019-10-18T00:40:59.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:59.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:40:59.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:40:59.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:40:59.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:40:59.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:00.674 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:41:00.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:00.670727+0000) 2019-10-18T00:41:00.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:00.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:00.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:00.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:01.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:01.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:01.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:01.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:02.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:41:02.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:02.627911+0000) 2019-10-18T00:41:02.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:02.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:02.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:02.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:03.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:03.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:03.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:03.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:04.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:04.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:04.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:04.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:04.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:04.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:05.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:05.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:05.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:05.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:06.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:41:06.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:06.081884+0000) 2019-10-18T00:41:06.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:06.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:06.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:06.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:07.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:07.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:07.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:07.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:08.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:08.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:08.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:08.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:09.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:41:09.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:09.162140+0000) 2019-10-18T00:41:09.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:09.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:09.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:09.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:09.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:09.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:10.674 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:41:10.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:10.670910+0000) 2019-10-18T00:41:10.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:10.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:10.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:10.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:11.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:11.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:11.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:11.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:12.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:41:12.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:12.628018+0000) 2019-10-18T00:41:12.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:12.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:12.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:12.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:13.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:13.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:13.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:13.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:14.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:14.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:14.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:14.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:14.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:14.919+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:15.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:15.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:15.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:15.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:16.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:41:16.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:16.081995+0000) 2019-10-18T00:41:16.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:16.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:16.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:17.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:17.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:17.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:17.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:18.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:18.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:18.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:18.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:19.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:41:19.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:19.162374+0000) 2019-10-18T00:41:19.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:19.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:19.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:19.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:19.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:19.919+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:20.674 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:41:20.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:20.671075+0000) 2019-10-18T00:41:20.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:20.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:20.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:20.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:21.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:21.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:21.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:21.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:22.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:41:22.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:22.628128+0000) 2019-10-18T00:41:22.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:22.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:22.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:22.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:23.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:23.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:23.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:23.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:24.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:24.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:24.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:24.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:24.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:24.919+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:25.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:25.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:25.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:25.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:26.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:41:26.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:26.082096+0000) 2019-10-18T00:41:26.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:26.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:26.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:26.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:27.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:27.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:27.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:27.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:28.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:28.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:28.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:28.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:29.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:41:29.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:29.162573+0000) 2019-10-18T00:41:29.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:29.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:29.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:29.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:29.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:29.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:30.674 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:41:30.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:30.671254+0000) 2019-10-18T00:41:30.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:30.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:30.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:30.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:31.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:31.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:31.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:31.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:32.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:41:32.622+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:32.628251+0000) 2019-10-18T00:41:32.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:32.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:32.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:32.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:33.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:33.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:34.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:34.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:34.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:34.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:34.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:34.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:35.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:35.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:35.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:35.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:36.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:41:36.078+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:36.082197+0000) 2019-10-18T00:41:36.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:36.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:36.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:36.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:37.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:37.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:37.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:37.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:38.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:38.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:38.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:38.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:39.172 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:41:39.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:39.162756+0000) 2019-10-18T00:41:39.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:39.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:39.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:39.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:39.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:39.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:40.674 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:41:40.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:40.671446+0000) 2019-10-18T00:41:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:40.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:40.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:41.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:41.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:41.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:41.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:42.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:41:42.622+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:42.628360+0000) 2019-10-18T00:41:42.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:42.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:42.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:42.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:43.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:43.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:43.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:43.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:44.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:44.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:44.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:44.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:44.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:44.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:45.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:45.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:45.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:45.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:46.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:41:46.078+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:46.082303+0000) 2019-10-18T00:41:46.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:46.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:46.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:46.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:47.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:47.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:47.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:47.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:48.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:48.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:48.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:48.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:49.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:41:49.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:49.162969+0000) 2019-10-18T00:41:49.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:49.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:49.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:49.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:49.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:49.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:50.674 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:41:50.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:50.671618+0000) 2019-10-18T00:41:50.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:50.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:50.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:50.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:51.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:51.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:51.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:51.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:52.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:41:52.622+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:52.628473+0000) 2019-10-18T00:41:52.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:52.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:52.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:52.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:53.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:53.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:53.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:53.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:54.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:54.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:54.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:54.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:54.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:54.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:41:55.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:55.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:55.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:55.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:56.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:41:56.078+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:56.082400+0000) 2019-10-18T00:41:56.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:56.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:56.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:56.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:57.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:57.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:57.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:57.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:58.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:58.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:58.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:58.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:59.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:41:59.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:41:59.163169+0000) 2019-10-18T00:41:59.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:59.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:41:59.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:41:59.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:41:59.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:41:59.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:00.675 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:42:00.669+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:00.671793+0000) 2019-10-18T00:42:00.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:00.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:00.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:00.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:01.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:01.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:01.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:01.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:02.631 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:42:02.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:02.628564+0000) 2019-10-18T00:42:02.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:02.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:02.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:02.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:03.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:03.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:03.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:03.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:04.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:04.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:04.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:04.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:04.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:04.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:05.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:05.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:05.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:05.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:06.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:42:06.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:06.082499+0000) 2019-10-18T00:42:06.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:06.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:06.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:06.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:07.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:07.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:07.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:07.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:08.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:08.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:08.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:08.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:09.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:09.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:09.163340+0000) 2019-10-18T00:42:09.167 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:09.161+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:09.164566+0000) 2019-10-18T00:42:09.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:09.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:09.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:09.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:09.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:09.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:10.675 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:42:10.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:10.671985+0000) 2019-10-18T00:42:10.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:10.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:10.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:10.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:11.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:11.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:11.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:11.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:12.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:42:12.621+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:12.628672+0000) 2019-10-18T00:42:12.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:12.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:12.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:12.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:13.667 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:13.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:13.663763+0000) 2019-10-18T00:42:13.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:13.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:13.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:13.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:14.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:14.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:14.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:14.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:14.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:14.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:15.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:15.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:15.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:15.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:16.085 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:42:16.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:16.082604+0000) 2019-10-18T00:42:16.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:16.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:16.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:16.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:17.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:17.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:17.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:17.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:18.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:18.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:18.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:18.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:19.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:19.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:19.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:19.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:19.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:19.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:20.675 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:42:20.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:20.672173+0000) 2019-10-18T00:42:20.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:20.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:20.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:20.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:21.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:21.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:21.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:21.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:22.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:42:22.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:22.628780+0000) 2019-10-18T00:42:22.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:22.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:22.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:22.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:23.667 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:23.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:23.663941+0000) 2019-10-18T00:42:23.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:23.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:23.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:23.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:24.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:24.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:24.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:24.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:24.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:24.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:25.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:25.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:25.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:25.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:26.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:42:26.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:26.082698+0000) 2019-10-18T00:42:26.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:26.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:26.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:26.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:27.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:27.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:27.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:27.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:28.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:28.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:28.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:28.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:29.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:29.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:29.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:29.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:29.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:29.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:30.675 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:42:30.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:30.672354+0000) 2019-10-18T00:42:30.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:30.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:30.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:30.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:31.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:31.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:31.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:31.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:32.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:42:32.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:32.628896+0000) 2019-10-18T00:42:32.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:32.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:32.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:32.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:33.668 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:33.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:33.664122+0000) 2019-10-18T00:42:33.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:33.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:33.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:33.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:34.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:34.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:34.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:34.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:34.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:34.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:35.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:35.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:35.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:35.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:36.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:42:36.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:36.082796+0000) 2019-10-18T00:42:36.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:36.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:36.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:36.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:37.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:37.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:37.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:37.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:38.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:38.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:38.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:38.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:39.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:39.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:39.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:39.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:39.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:39.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:40.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:42:40.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:40.672561+0000) 2019-10-18T00:42:40.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:40.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:40.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:40.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:41.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:41.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:41.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:41.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:42.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:42:42.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:42.629003+0000) 2019-10-18T00:42:42.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:42.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:42.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:42.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:43.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:43.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:43.664339+0000) 2019-10-18T00:42:43.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:43.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:43.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:43.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:44.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:44.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:44.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:44.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:44.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:44.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:45.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:45.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:45.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:45.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:46.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:42:46.080+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:46.082894+0000) 2019-10-18T00:42:46.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:46.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:46.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:46.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:47.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:47.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:47.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:47.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:48.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:48.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:48.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:48.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:49.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:49.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:49.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:49.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:49.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:49.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:50.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:50.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:50.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:50.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:50.675 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:42:50.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:50.672750+0000) 2019-10-18T00:42:51.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:51.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:51.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:51.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:52.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:42:52.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:52.629109+0000) 2019-10-18T00:42:52.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:52.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:52.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:52.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:53.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:53.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:53.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:53.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:53.667 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:42:53.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:53.664521+0000) 2019-10-18T00:42:54.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:54.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:54.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:54.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:54.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:54.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:42:55.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:55.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:55.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:55.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:56.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:42:56.080+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:42:56.082997+0000) 2019-10-18T00:42:56.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:56.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:56.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:56.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:57.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:57.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:32:28.965644+0000 (oldest deadline 2019-10-18T00:32:48.965644+0000) 2019-10-18T00:42:57.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:57.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:58.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:58.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:59.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:42:59.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:42:59.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:42:59.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:00.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:43:00.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:00.672948+0000) 2019-10-18T00:43:00.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:00.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:01.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:01.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:02.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:43:02.624+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:02.629207+0000) 2019-10-18T00:43:02.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:02.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:03.667 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:43:03.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:03.664702+0000) 2019-10-18T00:43:03.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:03.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:04.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:04.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:04.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:04.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:05.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:05.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:06.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:43:06.080+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:06.083097+0000) 2019-10-18T00:43:06.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:06.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:07.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:07.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:08.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:08.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:09.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:09.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:09.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:09.940+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:10.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:10.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:10.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:43:10.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:10.673134+0000) 2019-10-18T00:43:11.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:11.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:12.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:43:12.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:12.629307+0000) 2019-10-18T00:43:12.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:12.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:13.667 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:43:13.660+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:13.664853+0000) 2019-10-18T00:43:13.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:13.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:14.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:14.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:14.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:14.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:15.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:15.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:16.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:43:16.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:16.083204+0000) 2019-10-18T00:43:16.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:16.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:17.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:17.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:18.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:18.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:19.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:19.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:19.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:19.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:20.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:43:20.668+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:20.673329+0000) 2019-10-18T00:43:20.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:20.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:21.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:21.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:22.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:43:22.623+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:22.629413+0000) 2019-10-18T00:43:22.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:22.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:22.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:22.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:23.668 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:43:23.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:23.665031+0000) 2019-10-18T00:43:23.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:23.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:23.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:23.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:24.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:24.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:24.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:24.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:24.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:24.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:25.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:25.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:25.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:25.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:26.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:43:26.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:26.083306+0000) 2019-10-18T00:43:26.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:26.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:26.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:26.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:27.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:27.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:27.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:27.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:28.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:28.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:28.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:28.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:29.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:29.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:29.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:29.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:29.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:29.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:30.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:30.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:30.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:30.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:30.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:43:30.667+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:30.673511+0000) 2019-10-18T00:43:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:31.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:31.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:32.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:32.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:32.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:43:32.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:32.629522+0000) 2019-10-18T00:43:33.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:33.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:33.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:33.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:33.668 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:43:33.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:33.665180+0000) 2019-10-18T00:43:34.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:34.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:34.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:34.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:34.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:34.947+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:35.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:35.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:35.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:36.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:43:36.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:36.083409+0000) 2019-10-18T00:43:36.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:36.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:36.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:36.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:37.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:37.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:37.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:37.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:38.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:38.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:38.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:38.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:39.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:39.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:39.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:39.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:39.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:39.947+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:40.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:40.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:40.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:40.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:40.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:43:40.667+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:40.673698+0000) 2019-10-18T00:43:41.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:41.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:41.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:41.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:42.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:42.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:42.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:42.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:42.632 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:43:42.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:42.629623+0000) 2019-10-18T00:43:43.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:43.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:43.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:43.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:43.668 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:43:43.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:43.665359+0000) 2019-10-18T00:43:44.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:44.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:44.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:44.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:44.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:44.947+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:45.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:45.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:45.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:45.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:46.086 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:43:46.079+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:46.083512+0000) 2019-10-18T00:43:46.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:46.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:46.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:46.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:47.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:47.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:47.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:47.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:48.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:48.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:48.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:48.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:49.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:49.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:49.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:49.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:49.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:49.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:50.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:50.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:50.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:50.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:50.676 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:43:50.667+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:50.673918+0000) 2019-10-18T00:43:51.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:51.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:51.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:51.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:52.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:52.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:52.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:52.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:52.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:43:52.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:52.629723+0000) 2019-10-18T00:43:53.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:53.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:53.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:53.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:53.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:43:53.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:53.665542+0000) 2019-10-18T00:43:54.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:54.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:54.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:54.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:54.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:54.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:43:55.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:55.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:55.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:55.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:56.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:43:56.078+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:43:56.083617+0000) 2019-10-18T00:43:56.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:56.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:56.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:56.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:57.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:57.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:57.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:57.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:58.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:58.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:58.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:58.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:59.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:59.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:43:59.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:43:59.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:43:59.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:43:59.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:00.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:00.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:00.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:00.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:00.677 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:44:00.667+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:00.674125+0000) 2019-10-18T00:44:01.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:01.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:01.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:01.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:02.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:02.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:02.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:02.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:02.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:44:02.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:02.629821+0000) 2019-10-18T00:44:03.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:03.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:03.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:03.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:03.668 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:44:03.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:03.665726+0000) 2019-10-18T00:44:04.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:04.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:04.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:04.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:04.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:04.954+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:05.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:05.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:05.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:05.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:06.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:44:06.078+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:06.083717+0000) 2019-10-18T00:44:06.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:06.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:06.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:06.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:07.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:07.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:07.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:07.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:08.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:08.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:08.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:08.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:09.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:09.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:09.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:09.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:09.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:09.954+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:10.677 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:44:10.671+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:10.674340+0000) 2019-10-18T00:44:10.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:10.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:10.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:10.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:11.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:11.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:11.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:11.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:12.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:44:12.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:12.629923+0000) 2019-10-18T00:44:12.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:12.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:12.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:12.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:13.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:44:13.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:13.665939+0000) 2019-10-18T00:44:13.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:13.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:13.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:13.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:14.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:14.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:14.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:14.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:14.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:14.954+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:15.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:15.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:15.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:15.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:16.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:44:16.078+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:16.083821+0000) 2019-10-18T00:44:16.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:16.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:16.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:16.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:17.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:17.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:17.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:17.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:18.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:18.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:18.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:18.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:19.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:19.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:19.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:19.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:19.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:19.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:20.677 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:44:20.671+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:20.674485+0000) 2019-10-18T00:44:20.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:20.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:20.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:20.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:21.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:21.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:21.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:21.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:22.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:44:22.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:22.630028+0000) 2019-10-18T00:44:22.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:22.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:22.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:22.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:23.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:44:23.659+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:23.666119+0000) 2019-10-18T00:44:23.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:23.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:23.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:23.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:24.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:24.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:24.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:24.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:24.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:24.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:25.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:25.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:25.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:25.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:26.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:44:26.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:26.083922+0000) 2019-10-18T00:44:26.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:26.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:26.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:26.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:27.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:27.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:27.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:27.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:28.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:28.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:28.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:28.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:29.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:29.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:29.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:29.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:29.966 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:29.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:30.677 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:44:30.671+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:30.674648+0000) 2019-10-18T00:44:30.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:30.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:30.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:30.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:31.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:31.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:31.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:31.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:32.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:44:32.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:32.630127+0000) 2019-10-18T00:44:32.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:32.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:32.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:32.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:33.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:44:33.662+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:33.666337+0000) 2019-10-18T00:44:33.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:33.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:33.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:33.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:34.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:34.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:34.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:34.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:34.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:34.961+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:35.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:35.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:35.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:35.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:36.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:44:36.077+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:36.084017+0000) 2019-10-18T00:44:36.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:36.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:36.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:36.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:37.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:37.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:37.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:37.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:38.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:38.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:38.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:38.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:39.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:39.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:39.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:39.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:39.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:39.961+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:40.677 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:44:40.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:40.674808+0000) 2019-10-18T00:44:40.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:40.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:40.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:40.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:41.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:41.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:41.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:41.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:42.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:44:42.629+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:42.630233+0000) 2019-10-18T00:44:42.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:42.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:42.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:42.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:43.670 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:44:43.662+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:43.666548+0000) 2019-10-18T00:44:43.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:43.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:43.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:43.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:44.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:44.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:44.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:44.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:44.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:44.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:45.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:45.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:45.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:45.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:46.087 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:44:46.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:46.084113+0000) 2019-10-18T00:44:46.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:46.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:46.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:46.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:47.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:47.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:47.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:47.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:48.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:48.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:48.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:48.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:49.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:49.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:49.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:49.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:49.972 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:49.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:50.677 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:44:50.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:50.674986+0000) 2019-10-18T00:44:50.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:50.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:50.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:50.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:51.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:51.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:51.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:51.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:52.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:44:52.629+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:52.630331+0000) 2019-10-18T00:44:52.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:52.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:52.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:52.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:53.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:44:53.662+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:53.666693+0000) 2019-10-18T00:44:54.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:54.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:54.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:54.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:54.972 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:54.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:44:54.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:54.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:54.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:54.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:56.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:55.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:56.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:55.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:56.088 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:44:56.085+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:44:56.085511+0000) 2019-10-18T00:44:57.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:57.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:57.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:57.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:58.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:58.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:58.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:58.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:59.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:59.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:44:59.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:44:59.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:44:59.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:44:59.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:00.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:00.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:00.678 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:45:00.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:00.675174+0000) 2019-10-18T00:45:01.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:01.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:01.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:01.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:02.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:02.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:02.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:02.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:02.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:45:02.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:02.630427+0000) 2019-10-18T00:45:03.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:03.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:03.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:03.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:03.669 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:45:03.662+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:03.666879+0000) 2019-10-18T00:45:04.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:04.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:04.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:04.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:04.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:04.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:05.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:05.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:05.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:05.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:06.088 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:45:06.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:06.085609+0000) 2019-10-18T00:45:06.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:06.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:06.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:06.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:07.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:07.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:07.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:07.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:08.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:08.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:08.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:08.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:09.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:09.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:09.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:09.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:09.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:09.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:10.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:10.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:10.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:10.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:10.678 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:45:10.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:10.675356+0000) 2019-10-18T00:45:11.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:11.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:11.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:11.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:12.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:12.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:12.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:12.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:12.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:45:12.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:12.630534+0000) 2019-10-18T00:45:13.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:13.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:13.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:13.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:13.670 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:45:13.662+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:13.667094+0000) 2019-10-18T00:45:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:14.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:14.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:14.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:14.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:15.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:15.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:15.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:15.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:16.088 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:45:16.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:16.085710+0000) 2019-10-18T00:45:16.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:16.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:16.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:16.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:17.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:17.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:17.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:17.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:18.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:18.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:18.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:18.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:19.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:19.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:19.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:19.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:20.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:20.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:20.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:20.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:20.678 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:45:20.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:20.675550+0000) 2019-10-18T00:45:21.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:21.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:21.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:21.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:22.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:22.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:22.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:22.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:22.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:45:22.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:22.630641+0000) 2019-10-18T00:45:23.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:23.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:23.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:23.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:23.670 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:45:23.662+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:23.667284+0000) 2019-10-18T00:45:24.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:24.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:24.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:24.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:24.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:24.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:25.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:25.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:25.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:25.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:26.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:45:26.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:26.085813+0000) 2019-10-18T00:45:26.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:26.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:26.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:26.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:27.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:27.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:27.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:27.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:28.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:28.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:28.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:28.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:29.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:29.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:29.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:29.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:29.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:29.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:30.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:30.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:30.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:30.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:30.678 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:45:30.670+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:30.675737+0000) 2019-10-18T00:45:31.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:31.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:31.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:31.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:32.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:32.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:32.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:32.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:32.633 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:45:32.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:32.630744+0000) 2019-10-18T00:45:33.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:33.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:33.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:33.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:33.670 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:45:33.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:33.667494+0000) 2019-10-18T00:45:34.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:34.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:34.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:34.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:34.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:34.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:35.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:35.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:35.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:35.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:36.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:45:36.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:36.085917+0000) 2019-10-18T00:45:36.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:36.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:36.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:36.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:37.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:37.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:38.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:38.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:38.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:38.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:39.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:39.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:39.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:39.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:39.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:39.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:40.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:40.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:40.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:40.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:40.678 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:45:40.674+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:40.675916+0000) 2019-10-18T00:45:41.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:41.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:41.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:41.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:42.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:42.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:42.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:42.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:42.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:45:42.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:42.630848+0000) 2019-10-18T00:45:43.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:43.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:43.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:43.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:43.670 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:45:43.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:43.667706+0000) 2019-10-18T00:45:44.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:44.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:44.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:44.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:44.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:44.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:45.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:45.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:45.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:45.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:46.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:45:46.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:46.086018+0000) 2019-10-18T00:45:46.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:46.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:46.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:46.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:47.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:47.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:47.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:47.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:48.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:48.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:48.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:48.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:49.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:49.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:49.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:49.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:49.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:49.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:50.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:50.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:50.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:50.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:50.679 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:45:50.673+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:50.676105+0000) 2019-10-18T00:45:51.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:51.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:51.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:51.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:52.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:52.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:52.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:52.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:52.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:45:52.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:52.630952+0000) 2019-10-18T00:45:53.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:53.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:53.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:53.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:53.670 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:45:53.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:53.667924+0000) 2019-10-18T00:45:54.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:54.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:54.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:54.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:54.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:54.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:45:55.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:55.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:55.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:55.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:56.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:45:56.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:45:56.086123+0000) 2019-10-18T00:45:56.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:56.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:56.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:56.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:57.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:57.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:57.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:57.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:58.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:58.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:58.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:58.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:59.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:59.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:45:59.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:45:59.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:45:59.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:45:59.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:00.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:00.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:00.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:00.679 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:00.673+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:00.676293+0000) 2019-10-18T00:46:01.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:01.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:01.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:01.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:02.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:02.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:02.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:02.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:02.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:46:02.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:02.631053+0000) 2019-10-18T00:46:03.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:03.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:03.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:03.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:03.671 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:46:03.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:03.668134+0000) 2019-10-18T00:46:04.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:04.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:04.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:04.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:04.990 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:04.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:05.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:05.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:05.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:05.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:06.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:46:06.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:06.086217+0000) 2019-10-18T00:46:06.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:06.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:06.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:06.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:07.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:07.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:07.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:07.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:08.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:08.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:08.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:08.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:09.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:09.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:09.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:09.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:09.990 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:09.982+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:10.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:10.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:10.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:10.679 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:10.673+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:10.676465+0000) 2019-10-18T00:46:11.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:11.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:11.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:11.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:12.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:12.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:12.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:12.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:12.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:46:12.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:12.631159+0000) 2019-10-18T00:46:13.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:13.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:13.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:13.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:13.671 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:46:13.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:13.668316+0000) 2019-10-18T00:46:14.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:14.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:14.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:14.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:14.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:14.982+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:15.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:15.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:16.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:46:16.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:16.086317+0000) 2019-10-18T00:46:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:16.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:16.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:17.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:17.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:17.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:17.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:18.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:18.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:18.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:18.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:19.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:19.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:19.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:19.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:19.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:19.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:20.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:20.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:20.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:20.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:20.679 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:20.673+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:20.676644+0000) 2019-10-18T00:46:20.680 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:20.673+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:20.677319+0000) 2019-10-18T00:46:21.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:21.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:21.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:21.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:22.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:22.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:22.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:22.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:22.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:46:22.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:22.631264+0000) 2019-10-18T00:46:23.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:23.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:23.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:23.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:23.671 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:46:23.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:23.668496+0000) 2019-10-18T00:46:24.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:24.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:24.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:24.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:24.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:24.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:25.180 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:25.173+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:25.177035+0000) 2019-10-18T00:46:25.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:25.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:25.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:25.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:26.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:46:26.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:26.086418+0000) 2019-10-18T00:46:26.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:26.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:26.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:26.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:27.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:27.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:27.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:27.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:28.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:28.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:28.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:28.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:29.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:29.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:29.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:29.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:29.995 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:29.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:30.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:30.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:30.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:30.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:31.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:31.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:31.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:31.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:32.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:32.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:32.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:32.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:32.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:46:32.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:32.631367+0000) 2019-10-18T00:46:33.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:33.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:33.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:33.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:33.671 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:46:33.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:33.668679+0000) 2019-10-18T00:46:34.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:34.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:34.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:34.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:34.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:34.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:35.180 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:35.173+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:35.177198+0000) 2019-10-18T00:46:35.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:35.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:35.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:35.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:36.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:46:36.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:36.086515+0000) 2019-10-18T00:46:36.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:36.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:36.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:36.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:37.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:37.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:37.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:37.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:38.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:38.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:38.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:38.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:39.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:39.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:39.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:39.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:39.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:39.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:40.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:40.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:40.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:40.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:41.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:41.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:41.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:41.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:42.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:42.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:42.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:46:42.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:42.631475+0000) 2019-10-18T00:46:43.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:43.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:43.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:43.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:43.671 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:46:43.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:43.668894+0000) 2019-10-18T00:46:44.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:44.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:44.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:44.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:44.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:44.989+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:45.180 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:45.173+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:45.177385+0000) 2019-10-18T00:46:45.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:45.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:45.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:45.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:46.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:46:46.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:46.086610+0000) 2019-10-18T00:46:46.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:46.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:46.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:46.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:47.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:47.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:47.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:47.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:48.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:48.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:48.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:48.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:49.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:49.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:49.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:49.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:50.001 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:49.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:50.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:50.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:50.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:50.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:51.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:51.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:51.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:51.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:52.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:52.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:52.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:52.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:52.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:46:52.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:52.631582+0000) 2019-10-18T00:46:53.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:53.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:53.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:53.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:53.672 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:46:53.665+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:53.669074+0000) 2019-10-18T00:46:54.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:54.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:54.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:54.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:55.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:54.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:46:55.180 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:46:55.173+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:55.177532+0000) 2019-10-18T00:46:55.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:55.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:55.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:55.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:56.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:46:56.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:46:56.086708+0000) 2019-10-18T00:46:56.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:56.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:56.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:56.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:57.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:57.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:57.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:57.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:58.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:58.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:46:59.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:59.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:46:59.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:46:59.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:00.003 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:46:59.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:00.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:00.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:00.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:00.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:01.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:01.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:01.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:01.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:02.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:02.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:02.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:02.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:02.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:47:02.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:02.631696+0000) 2019-10-18T00:47:03.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:03.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:03.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:03.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:03.672 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:47:03.664+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:03.669255+0000) 2019-10-18T00:47:04.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:04.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:04.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:04.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:05.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:04.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:05.180 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:47:05.172+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:05.177711+0000) 2019-10-18T00:47:05.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:05.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:05.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:05.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:06.089 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:47:06.085+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:06.086812+0000) 2019-10-18T00:47:06.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:06.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:06.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:06.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:07.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:07.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:07.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:07.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:08.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:08.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:08.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:08.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:09.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:09.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:09.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:09.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:10.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:10.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:10.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:10.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:10.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:10.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:11.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:11.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:11.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:11.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:12.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:12.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:12.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:12.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:12.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:47:12.629+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:12.631814+0000) 2019-10-18T00:47:13.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:13.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:13.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:13.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:13.672 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:47:13.664+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:13.669436+0000) 2019-10-18T00:47:14.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:14.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:14.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:14.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:15.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:15.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:15.180 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:47:15.172+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:15.177886+0000) 2019-10-18T00:47:15.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:15.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:15.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:15.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:16.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:47:16.085+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:16.086912+0000) 2019-10-18T00:47:16.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:16.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:16.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:16.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:17.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:17.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:17.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:17.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:18.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:18.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:18.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:18.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:19.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:19.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:19.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:19.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:20.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:20.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:20.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:20.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:20.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:20.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:21.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:21.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:21.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:21.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:22.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:22.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:22.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:22.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:22.634 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:47:22.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:22.631916+0000) 2019-10-18T00:47:23.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:23.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:23.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:23.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:23.672 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:47:23.664+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:23.669618+0000) 2019-10-18T00:47:24.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:24.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:24.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:24.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:25.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:25.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:25.181 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:47:25.172+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:25.178074+0000) 2019-10-18T00:47:25.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:25.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:25.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:25.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:26.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:47:26.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:26.087013+0000) 2019-10-18T00:47:26.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:26.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:26.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:26.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:27.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:27.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:27.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:27.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:28.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:28.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:28.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:28.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:29.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:29.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:29.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:29.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:30.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:30.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:30.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:30.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:30.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:30.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:31.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:31.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:31.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:31.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:32.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:32.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:32.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:32.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:32.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:47:32.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:32.632020+0000) 2019-10-18T00:47:33.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:33.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:33.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:33.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:33.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:47:33.664+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:33.669799+0000) 2019-10-18T00:47:34.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:34.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:34.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:34.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:35.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:35.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:35.181 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:47:35.176+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:35.178261+0000) 2019-10-18T00:47:35.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:35.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:35.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:35.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:36.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:47:36.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:36.087117+0000) 2019-10-18T00:47:36.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:36.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:36.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:36.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:37.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:37.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:37.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:37.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:38.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:38.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:38.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:38.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:39.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:39.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:39.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:39.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:40.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:40.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:40.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:40.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:40.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:40.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:41.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:41.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:41.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:41.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:42.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:47:42.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:42.632128+0000) 2019-10-18T00:47:42.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:42.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:42.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:42.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:43.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:43.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:43.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:43.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:43.672 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:47:43.664+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:43.669988+0000) 2019-10-18T00:47:44.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:44.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:44.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:44.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:45.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:45.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:45.181 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:47:45.176+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:45.178403+0000) 2019-10-18T00:47:45.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:45.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:45.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:45.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:46.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:47:46.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:46.087217+0000) 2019-10-18T00:47:46.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:46.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:46.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:46.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:47.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:47.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:47.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:48.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:48.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:48.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:48.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:49.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:49.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:49.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:49.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:50.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:50.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:50.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:50.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:50.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:50.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:51.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:51.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:51.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:51.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:52.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:47:52.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:52.632226+0000) 2019-10-18T00:47:52.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:52.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:52.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:52.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:53.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:47:53.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:53.670175+0000) 2019-10-18T00:47:53.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:53.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:53.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:53.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:54.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:54.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:54.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:54.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:55.031 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:47:55.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:47:55.181 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:47:55.176+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:55.178551+0000) 2019-10-18T00:47:55.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:55.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:55.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:55.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:56.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:47:56.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:47:56.087319+0000) 2019-10-18T00:47:56.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:56.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:56.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:56.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:57.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:57.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:57.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:57.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:58.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:58.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:47:59.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:59.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:47:59.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:47:59.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:00.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:00.023+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:00.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:00.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:00.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:00.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:01.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:01.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:01.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:01.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:02.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:48:02.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:02.632329+0000) 2019-10-18T00:48:02.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:02.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:02.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:02.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:03.673 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:48:03.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:03.670385+0000) 2019-10-18T00:48:03.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:03.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:03.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:03.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:04.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:04.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:04.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:04.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:05.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:05.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:05.181 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:48:05.176+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:05.178691+0000) 2019-10-18T00:48:05.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:05.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:05.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:05.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:06.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:48:06.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:06.087416+0000) 2019-10-18T00:48:06.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:06.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:06.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:06.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:07.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:07.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:07.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:07.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:08.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:08.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:08.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:08.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:09.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:09.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:09.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:09.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:10.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:10.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:10.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:10.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:10.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:10.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:11.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:11.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:11.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:11.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:12.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:48:12.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:12.632439+0000) 2019-10-18T00:48:12.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:12.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:12.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:12.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:13.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:13.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:13.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:13.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:13.673 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:48:13.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:13.670567+0000) 2019-10-18T00:48:14.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:14.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:14.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:14.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:15.035 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:15.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:15.181 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:48:15.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:15.178869+0000) 2019-10-18T00:48:15.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:15.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:15.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:15.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:16.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:48:16.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:16.087523+0000) 2019-10-18T00:48:16.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:16.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:16.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:16.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:17.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:17.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:17.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:17.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:18.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:18.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:18.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:18.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:19.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:19.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:19.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:19.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:20.035 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:20.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:20.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:20.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:20.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:20.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:21.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:21.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:21.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:21.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:22.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:22.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:22.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:22.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:22.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:48:22.627+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:22.632544+0000) 2019-10-18T00:48:23.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:23.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:23.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:23.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:23.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:48:23.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:23.670752+0000) 2019-10-18T00:48:24.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:24.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:24.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:24.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:25.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:25.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:25.182 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:48:25.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:25.179061+0000) 2019-10-18T00:48:25.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:25.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:25.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:25.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:26.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:48:26.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:26.087625+0000) 2019-10-18T00:48:26.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:26.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:26.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:26.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:27.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:27.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:27.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:27.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:28.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:28.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:28.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:28.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:29.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:29.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:29.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:29.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:30.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:30.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:30.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:30.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:30.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:30.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:31.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:31.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:31.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:31.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:32.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:32.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:32.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:32.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:32.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:48:32.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:32.632647+0000) 2019-10-18T00:48:33.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:33.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:33.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:33.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:33.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:48:33.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:33.670937+0000) 2019-10-18T00:48:34.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:34.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:34.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:34.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:35.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:35.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:35.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:48:35.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:35.179255+0000) 2019-10-18T00:48:35.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:35.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:35.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:35.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:36.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:48:36.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:36.087724+0000) 2019-10-18T00:48:36.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:36.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:36.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:36.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:37.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:37.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:37.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:37.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:38.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:38.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:38.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:38.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:39.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:39.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:39.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:39.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:40.039 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:40.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:40.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:40.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:40.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:40.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:41.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:41.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:41.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:41.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:42.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:42.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:42.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:42.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:42.635 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:48:42.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:42.632757+0000) 2019-10-18T00:48:43.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:43.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:43.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:43.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:43.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:48:43.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:43.671119+0000) 2019-10-18T00:48:44.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:44.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:44.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:44.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:45.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:45.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:45.182 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:48:45.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:45.179444+0000) 2019-10-18T00:48:45.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:45.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:45.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:45.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:46.090 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:48:46.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:46.087831+0000) 2019-10-18T00:48:46.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:46.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:46.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:46.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:47.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:47.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:47.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:47.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:48.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:48.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:48.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:48.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:49.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:49.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:49.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:49.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:50.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:50.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:50.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:50.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:50.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:50.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 216 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:51.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:51.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:51.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:51.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:52.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:52.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:52.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:52.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:52.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:48:52.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:52.632865+0000) 2019-10-18T00:48:53.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:53.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:53.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:53.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:53.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:48:53.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:53.671308+0000) 2019-10-18T00:48:54.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:54.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:54.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:54.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:55.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:48:55.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:48:55.182 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:48:55.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:55.179643+0000) 2019-10-18T00:48:55.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:55.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:55.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:55.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:56.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:48:56.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:48:56.087937+0000) 2019-10-18T00:48:56.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:56.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:56.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:56.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:57.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:57.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:57.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:57.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:58.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:58.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:58.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:58.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:48:59.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:59.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:48:59.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:48:59.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:00.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:00.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:00.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:00.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:00.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:00.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:01.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:01.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:01.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:01.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:02.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:02.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:02.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:02.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:02.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:02.626+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:02.632967+0000) 2019-10-18T00:49:03.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:03.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:03.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:03.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:03.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:49:03.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:03.671487+0000) 2019-10-18T00:49:04.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:04.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:04.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:04.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:05.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:05.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:05.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:49:05.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:05.179831+0000) 2019-10-18T00:49:05.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:05.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:05.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:05.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:06.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:49:06.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:06.088038+0000) 2019-10-18T00:49:06.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:06.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:06.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:06.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:07.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:07.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:07.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:07.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:08.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:08.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:08.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:08.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:09.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:09.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:09.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:09.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:10.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:10.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:10.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:10.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:10.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:10.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:11.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:11.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:11.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:11.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:12.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:12.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:12.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:12.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:12.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:12.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:12.633079+0000) 2019-10-18T00:49:13.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:13.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:13.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:13.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:13.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:49:13.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:13.671699+0000) 2019-10-18T00:49:14.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:14.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:14.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:14.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:15.045 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:15.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:15.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:49:15.175+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:15.180051+0000) 2019-10-18T00:49:15.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:15.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:15.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:15.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:16.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:49:16.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:16.088139+0000) 2019-10-18T00:49:16.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:16.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:16.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:16.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:17.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:17.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:17.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:17.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:18.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:18.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:18.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:18.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:19.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:19.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:19.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:19.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:20.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:20.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:20.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:20.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:20.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:20.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:21.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:21.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:21.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:21.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:22.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:22.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:22.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:22.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:22.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:22.625+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:22.633181+0000) 2019-10-18T00:49:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:23.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:23.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:23.674 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:49:23.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:23.671866+0000) 2019-10-18T00:49:24.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:24.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:24.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:24.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:25.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:25.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:25.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:49:25.174+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:25.180267+0000) 2019-10-18T00:49:25.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:25.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:25.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:25.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:26.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:49:26.081+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:26.088240+0000) 2019-10-18T00:49:26.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:26.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:26.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:26.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:27.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:27.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:27.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:27.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:28.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:28.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:28.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:28.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:29.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:29.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:29.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:29.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:30.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:30.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:30.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:30.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:30.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:30.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:31.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:31.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:31.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:31.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:32.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:32.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:32.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:32.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:32.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:32.629+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:32.633275+0000) 2019-10-18T00:49:33.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:33.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:33.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:33.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:33.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:49:33.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:33.672050+0000) 2019-10-18T00:49:34.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:34.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:34.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:34.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:35.048 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:35.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:35.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:49:35.174+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:35.180445+0000) 2019-10-18T00:49:35.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:35.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:35.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:35.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:36.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:49:36.085+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:36.088346+0000) 2019-10-18T00:49:36.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:36.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:36.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:36.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:37.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:37.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:37.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:37.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:38.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:38.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:38.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:38.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:39.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:39.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:39.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:39.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:40.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:40.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:40.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:40.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:40.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:40.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:41.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:41.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:41.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:41.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:42.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:42.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:42.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:42.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:42.636 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:42.628+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:42.633368+0000) 2019-10-18T00:49:42.637 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:42.632+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:42.634697+0000) 2019-10-18T00:49:43.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:43.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:43.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:43.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:43.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:49:43.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:43.672262+0000) 2019-10-18T00:49:44.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:44.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:44.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:44.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:45.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:45.044+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:45.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:49:45.174+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:45.180623+0000) 2019-10-18T00:49:45.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:45.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:45.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:45.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:46.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:49:46.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:46.088446+0000) 2019-10-18T00:49:46.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:46.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:46.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:46.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:47.136 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:47.132+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:47.133578+0000) 2019-10-18T00:49:47.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:47.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:47.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:47.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:48.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:48.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:48.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:48.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:49.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:49.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:49.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:49.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:50.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:50.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:50.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:50.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:50.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:50.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:51.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:51.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:51.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:51.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:52.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:52.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:52.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:52.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:53.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:53.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:53.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:53.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:53.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:49:53.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:53.672446+0000) 2019-10-18T00:49:54.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:54.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:54.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:54.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:55.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:49:55.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:49:55.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:49:55.174+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:55.180798+0000) 2019-10-18T00:49:55.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:55.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:55.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:55.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:56.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:49:56.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:56.088554+0000) 2019-10-18T00:49:56.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:56.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:56.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:56.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:57.136 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:49:57.132+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:49:57.133679+0000) 2019-10-18T00:49:57.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:57.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:57.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:57.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:58.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:58.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:58.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:58.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:49:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:59.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:49:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:49:59.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:00.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:00.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:00.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:00.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:00.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:00.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:01.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:01.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:01.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:01.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:02.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:02.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:02.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:02.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:03.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:03.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:03.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:03.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:03.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:50:03.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:03.672656+0000) 2019-10-18T00:50:04.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:04.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:04.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:04.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:05.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:05.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:05.183 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:50:05.174+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:05.180980+0000) 2019-10-18T00:50:05.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:05.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:05.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:05.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:06.093 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:50:06.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:06.088658+0000) 2019-10-18T00:50:06.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:06.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:06.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:06.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:07.136 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:50:07.132+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:07.133775+0000) 2019-10-18T00:50:07.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:07.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:07.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:07.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:08.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:08.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:08.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:08.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:09.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:09.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:09.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:09.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:10.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:10.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:10.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:10.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:10.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:10.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:11.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:11.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:11.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:11.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:12.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:12.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:12.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:12.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:13.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:13.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:13.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:13.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:13.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:50:13.666+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:13.672866+0000) 2019-10-18T00:50:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:14.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:14.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:14.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:15.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:15.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:15.184 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:50:15.174+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:15.181165+0000) 2019-10-18T00:50:15.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:15.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:15.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:15.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:16.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:50:16.084+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:16.088761+0000) 2019-10-18T00:50:16.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:16.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:16.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:16.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:17.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:50:17.132+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:17.133877+0000) 2019-10-18T00:50:17.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:17.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:17.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:17.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:18.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:18.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:18.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:18.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:19.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:19.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:19.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:19.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:20.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:20.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:20.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:20.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:20.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:20.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:21.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:21.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:21.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:21.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:22.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:22.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:22.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:22.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:23.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:23.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:23.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:23.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:23.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:50:23.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:23.673051+0000) 2019-10-18T00:50:24.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:24.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:24.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:24.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:25.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:25.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:25.184 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:50:25.178+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:25.181353+0000) 2019-10-18T00:50:25.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:25.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:25.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:25.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:26.092 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:50:26.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:26.088871+0000) 2019-10-18T00:50:26.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:26.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:26.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:26.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:27.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:50:27.131+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:27.133981+0000) 2019-10-18T00:50:27.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:27.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:27.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:27.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:28.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:28.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:28.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:28.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:29.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:29.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:29.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:29.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:30.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:30.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:30.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:30.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:30.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:30.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:31.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:31.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:31.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:31.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:32.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:32.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:33.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:33.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:33.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:33.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:33.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:50:33.669+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:33.673236+0000) 2019-10-18T00:50:34.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:34.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:34.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:34.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:35.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:35.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:35.184 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:50:35.177+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:35.181560+0000) 2019-10-18T00:50:35.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:35.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:35.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:35.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:36.091 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:50:36.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:36.088974+0000) 2019-10-18T00:50:36.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:36.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:36.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:36.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:37.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:50:37.131+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:37.134075+0000) 2019-10-18T00:50:37.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:37.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:37.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:37.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:38.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:38.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:38.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:38.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:39.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:39.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:39.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:39.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:40.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:40.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:40.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:40.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:40.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:40.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:41.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:41.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:41.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:41.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:42.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:42.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:42.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:42.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:43.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:43.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:43.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:43.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:43.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:50:43.669+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:43.673449+0000) 2019-10-18T00:50:44.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:44.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:44.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:44.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:45.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:45.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:45.184 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:50:45.177+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:45.181767+0000) 2019-10-18T00:50:45.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:45.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:45.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:45.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:46.092 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:50:46.083+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:46.089080+0000) 2019-10-18T00:50:46.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:46.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:46.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:46.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:47.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:50:47.131+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:47.134180+0000) 2019-10-18T00:50:47.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:47.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:47.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:47.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:48.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:48.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:48.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:48.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:49.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:49.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:49.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:49.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:50.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:50.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:50.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:50.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:50.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:50.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:51.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:51.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:51.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:51.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:52.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:52.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:52.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:52.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:53.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:53.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:53.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:53.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:53.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:50:53.669+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:53.673620+0000) 2019-10-18T00:50:54.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:54.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:54.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:54.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:55.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:50:55.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:50:55.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:50:55.177+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:55.181964+0000) 2019-10-18T00:50:55.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:55.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:55.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:55.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:56.092 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:50:56.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:56.089190+0000) 2019-10-18T00:50:56.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:56.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:56.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:56.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:57.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:50:57.130+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:50:57.134286+0000) 2019-10-18T00:50:57.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:57.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:57.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:57.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:58.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:58.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:58.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:58.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:50:59.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:59.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:50:59.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:50:59.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:00.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:00.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:00.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:00.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:00.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:00.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:01.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:01.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:01.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:01.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:02.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:02.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:02.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:02.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:03.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:03.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:03.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:03.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:03.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:51:03.669+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:03.673830+0000) 2019-10-18T00:51:04.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:04.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:04.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:04.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:05.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:05.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:05.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:51:05.177+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:05.182127+0000) 2019-10-18T00:51:05.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:05.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:05.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:05.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:06.092 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:06.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:06.089288+0000) 2019-10-18T00:51:06.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:06.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:06.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:06.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:07.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:07.130+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:07.134386+0000) 2019-10-18T00:51:07.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:07.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:07.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:07.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:08.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:08.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:08.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:08.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:09.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:09.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:09.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:09.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:10.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:10.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:10.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:10.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:11.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:11.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:11.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:11.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:12.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:12.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:12.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:12.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:13.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:13.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:13.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:13.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:13.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:51:13.669+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:13.674044+0000) 2019-10-18T00:51:14.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:14.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:14.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:14.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:15.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:15.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:15.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:51:15.181+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:15.182272+0000) 2019-10-18T00:51:15.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:15.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:15.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:15.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:16.092 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:16.082+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:16.089386+0000) 2019-10-18T00:51:16.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:16.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:16.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:16.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:17.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:17.130+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:17.134493+0000) 2019-10-18T00:51:17.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:17.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:17.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:17.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:18.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:18.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:18.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:18.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:19.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:19.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:20.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:20.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:20.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:20.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:20.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:20.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:21.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:21.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:21.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:21.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:22.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:22.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:22.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:22.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:23.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:23.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:23.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:23.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:23.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:51:23.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:23.674236+0000) 2019-10-18T00:51:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:24.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:24.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:25.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:25.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:25.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:25.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:25.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:25.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:25.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:51:25.181+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:25.182452+0000) 2019-10-18T00:51:26.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:26.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:26.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:26.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:26.093 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:26.086+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:26.089484+0000) 2019-10-18T00:51:26.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:26.086+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:26.090697+0000) 2019-10-18T00:51:27.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:27.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:27.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:27.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:27.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:27.130+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:27.134598+0000) 2019-10-18T00:51:28.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:27.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:28.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:27.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:29.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:29.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:29.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:29.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:30.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:30.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:30.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:30.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:30.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:30.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:30.592 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:30.585+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:30.589679+0000) 2019-10-18T00:51:31.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:31.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:31.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:31.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:32.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:32.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:32.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:32.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:33.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:33.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:33.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:33.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:33.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:51:33.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:33.674414+0000) 2019-10-18T00:51:34.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:34.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:34.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:34.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:35.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:35.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:35.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:35.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:35.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:35.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:35.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:51:35.181+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:35.182653+0000) 2019-10-18T00:51:36.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:36.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:36.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:36.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:37.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:37.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:37.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:37.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:37.137 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:37.129+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:37.134704+0000) 2019-10-18T00:51:38.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:38.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:38.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:38.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:39.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:39.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:39.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:39.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:40.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:40.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:40.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:40.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:40.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:40.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:40.592 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:40.585+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:40.589782+0000) 2019-10-18T00:51:41.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:41.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:41.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:41.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:42.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:42.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:42.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:42.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:43.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:43.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:43.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:43.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:43.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:51:43.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:43.674618+0000) 2019-10-18T00:51:44.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:44.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:44.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:44.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:45.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:45.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:45.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:45.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:45.075 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:45.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:45.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:51:45.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:45.182869+0000) 2019-10-18T00:51:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:45.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:45.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:47.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:47.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:47.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:47.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:47.139 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:47.129+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:47.134802+0000) 2019-10-18T00:51:48.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:48.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:48.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:48.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:49.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:49.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:49.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:49.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:50.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:50.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:50.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:50.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:50.592 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:51:50.589+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:50.589896+0000) 2019-10-18T00:51:51.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:51.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:51.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:51.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:52.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:52.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:52.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:52.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:53.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:53.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:53.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:53.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:53.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:51:53.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:53.674801+0000) 2019-10-18T00:51:54.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:54.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:54.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:54.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:55.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:55.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:55.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:55.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:55.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:51:55.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:51:55.185 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:51:55.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:55.183046+0000) 2019-10-18T00:51:56.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:56.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:56.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:56.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:57.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:57.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:57.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:57.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:57.138 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:57.133+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:57.134900+0000) 2019-10-18T00:51:57.138 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:51:57.133+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:51:57.135997+0000) 2019-10-18T00:51:58.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:58.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:58.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:58.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:51:59.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:59.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:51:59.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:51:59.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:00.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:00.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:00.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:00.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:00.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:00.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:00.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:52:00.589+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:00.590007+0000) 2019-10-18T00:52:01.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:01.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:01.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:01.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:01.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:52:01.633+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:01.635097+0000) 2019-10-18T00:52:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:02.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:02.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:03.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:03.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:03.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:03.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:03.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:52:03.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:03.675014+0000) 2019-10-18T00:52:04.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:04.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:04.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:04.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:05.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:05.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:05.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:52:05.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:05.183233+0000) 2019-10-18T00:52:05.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:05.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:05.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:05.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:06.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:06.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:06.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:06.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:07.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:07.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:07.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:07.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:08.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:08.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:08.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:08.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:09.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:09.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:09.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:09.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:10.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:10.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:10.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:10.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:10.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:10.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:10.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:52:10.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:10.590114+0000) 2019-10-18T00:52:11.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:11.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:11.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:11.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:11.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:52:11.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:11.635203+0000) 2019-10-18T00:52:12.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:12.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:12.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:12.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:13.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:13.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:13.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:13.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:13.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:52:13.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:13.675227+0000) 2019-10-18T00:52:14.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:14.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:14.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:14.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:15.087 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:15.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:15.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:52:15.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:15.183402+0000) 2019-10-18T00:52:15.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:15.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:15.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:15.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:16.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:16.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:16.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:16.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:17.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:17.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:17.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:17.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:18.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:18.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:18.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:18.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:19.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:19.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:19.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:19.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:20.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:20.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:20.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:20.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:20.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:20.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:20.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:52:20.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:20.590216+0000) 2019-10-18T00:52:21.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:21.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:21.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:21.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:21.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:52:21.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:21.635303+0000) 2019-10-18T00:52:22.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:22.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:22.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:22.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:23.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:23.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:23.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:23.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:23.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:52:23.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:23.675372+0000) 2019-10-18T00:52:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:24.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:24.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:25.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:25.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:25.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:52:25.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:25.183568+0000) 2019-10-18T00:52:25.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:25.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:25.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:25.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:26.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:26.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:26.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:26.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:27.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:27.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:27.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:27.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:28.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:28.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:28.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:28.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:29.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:29.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:29.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:29.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:30.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:30.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:30.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:30.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:30.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:30.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:30.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:52:30.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:30.590324+0000) 2019-10-18T00:52:31.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:31.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:31.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:31.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:31.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:52:31.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:31.635413+0000) 2019-10-18T00:52:32.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:32.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:32.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:32.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:33.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:33.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:33.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:33.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:33.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:52:33.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:33.675582+0000) 2019-10-18T00:52:34.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:34.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:34.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:34.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:35.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:35.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:35.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:52:35.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:35.183731+0000) 2019-10-18T00:52:35.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:35.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:35.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:35.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:36.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:36.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:36.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:36.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:37.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:37.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:37.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:37.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:38.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:38.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:38.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:38.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:39.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:39.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:39.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:39.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:40.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:40.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:40.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:40.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:40.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:40.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:40.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:52:40.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:40.590432+0000) 2019-10-18T00:52:41.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:41.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:41.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:41.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:41.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:52:41.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:41.635519+0000) 2019-10-18T00:52:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:42.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:42.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:43.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:43.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:43.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:43.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:43.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:52:43.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:43.675791+0000) 2019-10-18T00:52:44.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:44.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:44.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:44.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:45.095 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:45.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:45.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:52:45.180+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:45.183905+0000) 2019-10-18T00:52:45.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:45.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:45.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:45.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:46.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:46.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:46.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:46.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:47.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:47.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:47.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:47.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:48.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:48.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:48.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:48.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:49.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:49.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:49.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:49.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:50.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:50.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:50.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:50.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:50.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:50.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:50.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:52:50.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:50.590538+0000) 2019-10-18T00:52:51.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:51.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:51.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:51.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:51.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:52:51.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:51.635616+0000) 2019-10-18T00:52:52.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:52.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:52.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:52.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:53.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:53.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:53.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:53.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:53.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:52:53.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:53.676001+0000) 2019-10-18T00:52:54.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:54.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:54.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:54.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:55.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:52:55.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:52:55.186 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:52:55.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:52:55.184084+0000) 2019-10-18T00:52:55.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:55.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:55.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:55.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:56.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:56.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:56.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:56.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:57.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:57.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:57.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:57.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:58.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:58.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:58.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:58.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:52:59.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:59.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:52:59.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:52:59.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:00.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:00.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:00.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:00.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:00.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:00.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:00.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:53:00.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:00.590641+0000) 2019-10-18T00:53:01.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:01.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:01.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:01.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:01.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:53:01.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:01.635722+0000) 2019-10-18T00:53:02.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:02.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:02.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:02.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:03.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:03.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:03.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:03.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:03.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:53:03.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:03.676214+0000) 2019-10-18T00:53:04.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:04.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:04.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:04.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:05.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:05.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:05.187 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:53:05.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:05.184267+0000) 2019-10-18T00:53:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:05.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:05.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:06.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:06.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:06.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:06.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:07.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:07.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:07.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:07.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:08.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:08.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:08.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:08.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:09.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:09.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:09.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:09.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:10.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:10.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:10.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:10.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:10.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:10.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:10.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:53:10.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:10.590750+0000) 2019-10-18T00:53:11.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:11.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:11.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:11.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:11.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:53:11.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:11.635826+0000) 2019-10-18T00:53:12.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:12.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:12.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:12.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:13.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:13.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:13.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:13.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:13.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:53:13.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:13.676385+0000) 2019-10-18T00:53:14.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:14.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:14.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:14.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:15.103 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:15.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:15.187 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:53:15.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:15.184431+0000) 2019-10-18T00:53:15.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:15.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:15.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:15.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:16.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:16.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:16.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:16.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:17.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:17.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:17.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:17.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:18.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:18.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:18.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:18.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:19.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:19.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:19.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:19.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:20.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:20.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:20.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:20.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:20.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:20.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:20.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:53:20.586+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:20.590863+0000) 2019-10-18T00:53:21.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:21.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:21.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:21.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:21.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:53:21.630+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:21.635927+0000) 2019-10-18T00:53:22.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:22.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:22.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:22.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:23.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:23.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:23.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:23.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:23.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:53:23.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:23.676565+0000) 2019-10-18T00:53:24.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:24.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:24.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:24.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:25.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:25.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:25.187 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:53:25.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:25.184596+0000) 2019-10-18T00:53:25.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:25.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:25.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:25.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:26.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:26.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:26.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:26.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:27.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:27.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:27.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:27.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:28.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:28.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:28.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:28.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:29.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:29.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:29.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:29.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:30.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:30.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:30.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:30.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:30.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:30.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:30.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:53:30.586+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:30.590976+0000) 2019-10-18T00:53:31.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:31.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:31.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:31.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:31.638 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:53:31.630+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:31.636036+0000) 2019-10-18T00:53:32.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:32.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:32.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:32.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:33.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:33.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:33.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:33.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:33.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:53:33.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:33.676750+0000) 2019-10-18T00:53:34.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:34.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:34.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:34.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:35.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:35.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:35.187 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:53:35.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:35.184756+0000) 2019-10-18T00:53:35.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:35.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:35.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:35.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:36.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:36.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:43:02.195108+0000 (oldest deadline 2019-10-18T00:43:22.195108+0000) 2019-10-18T00:53:36.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:36.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:37.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:37.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:38.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:38.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:39.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:39.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:40.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:40.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:40.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:40.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:40.593 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:53:40.586+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:40.591087+0000) 2019-10-18T00:53:41.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:41.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:41.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:53:41.630+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:41.636142+0000) 2019-10-18T00:53:42.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:42.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:43.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:43.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:43.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:53:43.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:43.676933+0000) 2019-10-18T00:53:44.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:44.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:45.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:45.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:45.187 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:53:45.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:45.184938+0000) 2019-10-18T00:53:45.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:45.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:46.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:46.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:47.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:47.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:48.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:48.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:49.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:49.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:50.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:50.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:50.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:50.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:50.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:53:50.586+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:50.591196+0000) 2019-10-18T00:53:51.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:51.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:51.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:53:51.629+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:51.636241+0000) 2019-10-18T00:53:52.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:52.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:53.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:53.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:53.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:53:53.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:53.677112+0000) 2019-10-18T00:53:54.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:54.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:55.114 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:53:55.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:53:55.188 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:53:55.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:53:55.185164+0000) 2019-10-18T00:53:55.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:55.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:56.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:56.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:57.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:57.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:58.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:58.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:53:59.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:53:59.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:00.114 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:00.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:00.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:00.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:00.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:54:00.585+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:00.591301+0000) 2019-10-18T00:54:01.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:01.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:01.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:54:01.629+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:01.636355+0000) 2019-10-18T00:54:02.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:02.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:02.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:02.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:03.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:03.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:03.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:03.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:03.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:54:03.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:03.677293+0000) 2019-10-18T00:54:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:04.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:04.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:05.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:05.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:05.188 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:54:05.179+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:05.185395+0000) 2019-10-18T00:54:05.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:05.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:05.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:05.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:06.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:06.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:06.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:06.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:07.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:07.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:07.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:07.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:08.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:08.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:09.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:09.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:09.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:09.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:10.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:10.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:10.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:10.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:10.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:10.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:10.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:54:10.585+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:10.591411+0000) 2019-10-18T00:54:11.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:11.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:11.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:11.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:11.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:54:11.633+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:11.636454+0000) 2019-10-18T00:54:12.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:12.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:12.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:12.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:13.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:13.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:13.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:13.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:13.680 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:54:13.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:13.677488+0000) 2019-10-18T00:54:14.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:14.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:14.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:14.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:15.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:15.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:15.188 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:54:15.178+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:15.185596+0000) 2019-10-18T00:54:15.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:15.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:15.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:15.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:16.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:16.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:16.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:16.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:17.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:17.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:17.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:17.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:18.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:18.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:18.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:18.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:19.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:19.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:19.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:20.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:20.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:20.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:20.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:20.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:20.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:20.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:54:20.589+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:20.591521+0000) 2019-10-18T00:54:21.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:21.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:21.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:21.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:21.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:54:21.633+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:21.636550+0000) 2019-10-18T00:54:22.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:22.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:22.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:22.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:23.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:23.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:23.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:23.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:23.680 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:54:23.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:23.677699+0000) 2019-10-18T00:54:24.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:24.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:24.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:24.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:25.120 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:25.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:25.188 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:54:25.178+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:25.185761+0000) 2019-10-18T00:54:25.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:25.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:25.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:25.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:26.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:26.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:26.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:26.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:27.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:27.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:27.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:27.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:28.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:28.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:28.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:28.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:29.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:29.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:29.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:29.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:30.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:30.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:30.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:54:30.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:30.591626+0000) 2019-10-18T00:54:30.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:30.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:30.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:30.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:31.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:31.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:31.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:31.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:31.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:54:31.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:31.636653+0000) 2019-10-18T00:54:32.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:32.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:32.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:32.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:33.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:33.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:33.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:33.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:33.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:54:33.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:33.677886+0000) 2019-10-18T00:54:34.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:34.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:34.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:34.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:35.122 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:35.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:35.188 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:54:35.178+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:35.185969+0000) 2019-10-18T00:54:35.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:35.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:35.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:35.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:36.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:36.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:36.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:36.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:37.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:37.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:37.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:37.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:38.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:38.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:38.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:38.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:39.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:39.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:39.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:39.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:40.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:40.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:40.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:54:40.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:40.591750+0000) 2019-10-18T00:54:40.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:40.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:40.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:40.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:41.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:54:41.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:41.636759+0000) 2019-10-18T00:54:41.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:41.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:41.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:41.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:42.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:42.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:42.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:42.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:43.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:54:43.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:43.678076+0000) 2019-10-18T00:54:43.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:43.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:43.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:43.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:44.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:44.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:44.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:44.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:45.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:45.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:45.189 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:54:45.182+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:45.186221+0000) 2019-10-18T00:54:45.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:45.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:45.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:45.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:46.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:46.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:46.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:46.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:47.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:47.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:47.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:47.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:48.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:48.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:48.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:48.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:49.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:49.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:49.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:49.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:50.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:50.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:50.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:54:50.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:50.591861+0000) 2019-10-18T00:54:50.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:50.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:50.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:50.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:51.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:51.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:51.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:51.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:51.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:54:51.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:51.636857+0000) 2019-10-18T00:54:52.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:52.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:52.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:52.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:53.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:53.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:53.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:53.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:53.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:54:53.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:53.678262+0000) 2019-10-18T00:54:54.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:54.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:54.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:54.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:55.126 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:54:55.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:54:55.189 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:54:55.182+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:54:55.186399+0000) 2019-10-18T00:54:55.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:55.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:55.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:55.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:56.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:56.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:56.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:56.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:57.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:57.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:57.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:57.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:58.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:58.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:58.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:58.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:54:59.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:59.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:54:59.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:54:59.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:00.127 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:00.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:00.594 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:00.588+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:00.591964+0000) 2019-10-18T00:55:00.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:00.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:00.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:00.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:01.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:55:01.632+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:01.636960+0000) 2019-10-18T00:55:01.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:01.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:01.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:01.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:02.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:02.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:02.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:02.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:03.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:55:03.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:03.678439+0000) 2019-10-18T00:55:03.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:03.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:03.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:03.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:04.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:04.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:04.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:04.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:05.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:05.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:05.189 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:55:05.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:05.186583+0000) 2019-10-18T00:55:05.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:05.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:05.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:05.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:06.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:06.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:06.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:06.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:07.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:07.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:07.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:07.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:08.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:08.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:08.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:08.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:09.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:09.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:09.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:09.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:10.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:10.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:10.595 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:10.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:10.592079+0000) 2019-10-18T00:55:10.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:10.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:10.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:10.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:11.639 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:55:11.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:11.637067+0000) 2019-10-18T00:55:11.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:11.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:11.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:11.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:12.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:12.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:12.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:12.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:13.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:55:13.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:13.678627+0000) 2019-10-18T00:55:13.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:13.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:13.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:13.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:15.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:15.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:15.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:15.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:15.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:15.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:15.189 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:55:15.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:15.186791+0000) 2019-10-18T00:55:16.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:16.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:16.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:16.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:17.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:17.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:17.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:17.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:18.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:18.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:18.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:18.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:19.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:19.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:19.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:19.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:20.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:20.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:20.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:20.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:20.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:20.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:20.595 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:20.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:20.592182+0000) 2019-10-18T00:55:21.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:21.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:21.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:21.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:21.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:55:21.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:21.637165+0000) 2019-10-18T00:55:22.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:22.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:22.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:22.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:23.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:23.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:23.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:23.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:23.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:55:23.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:23.678808+0000) 2019-10-18T00:55:24.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:24.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:24.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:24.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:25.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:25.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:25.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:25.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:25.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:25.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:25.189 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:55:25.185+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:25.186985+0000) 2019-10-18T00:55:26.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:26.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:26.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:26.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:27.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:27.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:27.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:27.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:28.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:28.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:28.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:28.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:29.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:29.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:29.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:29.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:30.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:30.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:30.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:30.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:30.134 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:30.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:30.596 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:30.587+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:30.592281+0000) 2019-10-18T00:55:30.596 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:30.587+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:30.593168+0000) 2019-10-18T00:55:31.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:31.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:31.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:31.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:31.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:55:31.631+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:31.637267+0000) 2019-10-18T00:55:32.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:32.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:32.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:32.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:33.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:33.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:33.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:33.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:33.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:55:33.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:33.678990+0000) 2019-10-18T00:55:34.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:34.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:34.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:34.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:35.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:35.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:35.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:35.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:35.095 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:35.087+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:35.092495+0000) 2019-10-18T00:55:35.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:35.131+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:35.190 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:55:35.185+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:35.187200+0000) 2019-10-18T00:55:36.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:36.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:36.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:36.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:37.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:37.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:37.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:37.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:38.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:38.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:38.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:38.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:39.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:39.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:39.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:39.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:40.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:40.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:40.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:40.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:40.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:40.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:41.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:41.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:41.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:55:41.630+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:41.637370+0000) 2019-10-18T00:55:42.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:42.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:42.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:42.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:42.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:42.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:42.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:42.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:43.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:55:43.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:43.679175+0000) 2019-10-18T00:55:43.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:43.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:43.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:43.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:44.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:44.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:44.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:44.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:45.095 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:45.086+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:45.092597+0000) 2019-10-18T00:55:45.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:45.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:45.191 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:55:45.185+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:45.187395+0000) 2019-10-18T00:55:45.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:45.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:45.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:45.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:46.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:46.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:46.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:46.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:47.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:47.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:47.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:47.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:48.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:48.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:48.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:48.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:49.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:49.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:49.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:49.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:50.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:50.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:50.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:50.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:50.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:50.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:51.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:55:51.634+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:51.637462+0000) 2019-10-18T00:55:51.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:51.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:51.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:51.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:53.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:53.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:53.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:53.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:53.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:55:53.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:53.679355+0000) 2019-10-18T00:55:54.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:54.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:54.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:54.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:54.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:54.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:54.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:54.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:55.095 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:55:55.086+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:55.092708+0000) 2019-10-18T00:55:55.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:55:55.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:55:55.190 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:55:55.185+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:55:55.187628+0000) 2019-10-18T00:55:55.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:55.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:55.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:55.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:56.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:56.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:56.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:56.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:57.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:57.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:57.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:57.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:58.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:58.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:58.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:58.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:55:59.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:59.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:55:59.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:55:59.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:00.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:00.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:00.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:00.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:00.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:00.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:01.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:56:01.634+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:01.637566+0000) 2019-10-18T00:56:01.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:01.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:01.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:01.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:02.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:02.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:02.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:02.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:03.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:56:03.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:03.679543+0000) 2019-10-18T00:56:03.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:03.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:03.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:03.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:04.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:04.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:04.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:04.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:05.095 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:56:05.086+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:05.092815+0000) 2019-10-18T00:56:05.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:05.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:05.190 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:05.185+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:05.187785+0000) 2019-10-18T00:56:05.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:05.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:05.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:05.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:06.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:06.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:06.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:06.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:07.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:07.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:07.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:07.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:08.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:08.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:08.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:08.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:09.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:09.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:09.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:09.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:10.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:10.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:10.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:10.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:10.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:10.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:11.646 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:56:11.634+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:11.637672+0000) 2019-10-18T00:56:11.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:11.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:11.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:11.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:12.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:12.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:12.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:12.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:13.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:56:13.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:13.679759+0000) 2019-10-18T00:56:13.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:13.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:13.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:13.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:14.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:14.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:14.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:14.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:15.095 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:56:15.085+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:15.092920+0000) 2019-10-18T00:56:15.147 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:15.137+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:15.190 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:15.185+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:15.187973+0000) 2019-10-18T00:56:15.191 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:15.185+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:15.189054+0000) 2019-10-18T00:56:15.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:15.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:15.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:15.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:16.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:16.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:16.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:16.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:17.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:17.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:17.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:17.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:18.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:18.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:18.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:18.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:19.691 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:19.685+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:19.688661+0000) 2019-10-18T00:56:19.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:19.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:19.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:19.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:20.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:20.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:20.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:20.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:20.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:20.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:21.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:56:21.633+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:21.637774+0000) 2019-10-18T00:56:21.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:21.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:21.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:21.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:22.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:22.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:22.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:22.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:23.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:56:23.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:23.679940+0000) 2019-10-18T00:56:23.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:23.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:23.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:23.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:24.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:24.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:24.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:24.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:25.095 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:56:25.085+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:25.093023+0000) 2019-10-18T00:56:25.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:25.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:25.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:25.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:25.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:25.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:26.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:26.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:26.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:26.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:27.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:27.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:27.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:27.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:28.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:28.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:28.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:28.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:29.691 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:29.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:29.688852+0000) 2019-10-18T00:56:29.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:29.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:29.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:29.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:30.150 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:30.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:30.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:30.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:30.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:30.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:31.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:56:31.633+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:31.637880+0000) 2019-10-18T00:56:31.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:31.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:31.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:31.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:32.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:32.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:32.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:32.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:33.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:56:33.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:33.680117+0000) 2019-10-18T00:56:33.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:33.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:33.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:33.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:34.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:34.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:34.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:34.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:35.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:56:35.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:35.093120+0000) 2019-10-18T00:56:35.151 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:35.145+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:35.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:35.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:35.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:35.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:36.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:36.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:36.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:36.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:37.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:37.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:37.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:37.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:38.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:38.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:38.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:38.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:39.692 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:39.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:39.689061+0000) 2019-10-18T00:56:39.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:39.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:39.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:39.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:40.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:40.145+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:40.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:40.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:40.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:40.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:41.640 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:56:41.637+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:41.637986+0000) 2019-10-18T00:56:41.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:41.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:41.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:41.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:42.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:42.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:42.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:42.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:43.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:56:43.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:43.680301+0000) 2019-10-18T00:56:43.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:43.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:43.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:43.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:44.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:44.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:45.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:56:45.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:45.093218+0000) 2019-10-18T00:56:45.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:45.149+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:45.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:45.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:45.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:45.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:46.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:46.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:46.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:46.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:47.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:47.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:47.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:47.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:48.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:48.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:48.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:48.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:49.692 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:49.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:49.689248+0000) 2019-10-18T00:56:49.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:49.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:49.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:49.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:50.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:50.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:50.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:50.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:50.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:50.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:51.641 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:56:51.636+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:51.638093+0000) 2019-10-18T00:56:51.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:51.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:51.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:51.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:52.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:52.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:52.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:52.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:53.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:56:53.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:53.680483+0000) 2019-10-18T00:56:53.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:53.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:53.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:53.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:54.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:54.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:54.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:54.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:55.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:56:55.088+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:55.093320+0000) 2019-10-18T00:56:55.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:56:55.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:56:55.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:55.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:55.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:55.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:56.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:56.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:56.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:56.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:57.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:57.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:57.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:57.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:58.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:58.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:58.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:58.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:56:59.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:56:59.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:56:59.689435+0000) 2019-10-18T00:56:59.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:59.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:56:59.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:56:59.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:00.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:00.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:00.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:00.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:00.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:00.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:01.641 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:01.636+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:01.638196+0000) 2019-10-18T00:57:01.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:01.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:01.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:01.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:02.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:02.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:02.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:02.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:03.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:57:03.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:03.680670+0000) 2019-10-18T00:57:03.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:03.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:03.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:03.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:04.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:04.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:04.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:04.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:05.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:57:05.088+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:05.093424+0000) 2019-10-18T00:57:05.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:05.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:05.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:05.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:05.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:05.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:06.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:06.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:06.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:06.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:07.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:07.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:07.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:07.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:08.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:08.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:08.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:08.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:09.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:57:09.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:09.689621+0000) 2019-10-18T00:57:09.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:09.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:09.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:09.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:10.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:10.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:10.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:10.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:10.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:10.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:11.641 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:11.636+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:11.638296+0000) 2019-10-18T00:57:11.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:11.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:11.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:11.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:12.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:12.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:12.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:12.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:13.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:57:13.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:13.680853+0000) 2019-10-18T00:57:13.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:13.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:13.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:13.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:14.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:14.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:14.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:14.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:15.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:57:15.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:15.093533+0000) 2019-10-18T00:57:15.158 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:15.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:15.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:15.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:15.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:15.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:16.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:16.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:17.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:17.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:17.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:17.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:18.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:18.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:18.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:18.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:19.692 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:57:19.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:19.689795+0000) 2019-10-18T00:57:19.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:19.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:19.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:19.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:20.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:20.156+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:20.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:20.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:20.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:20.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:21.641 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:21.636+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:21.638399+0000) 2019-10-18T00:57:21.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:21.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:21.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:21.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:22.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:22.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:22.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:23.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:57:23.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:23.681061+0000) 2019-10-18T00:57:23.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:23.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:23.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:23.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:24.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:24.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:24.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:24.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:25.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:57:25.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:25.093635+0000) 2019-10-18T00:57:25.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:25.155+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:25.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:25.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:25.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:25.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:26.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:26.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:26.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:26.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:27.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:27.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:27.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:27.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:28.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:28.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:28.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:28.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:29.692 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:57:29.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:29.689947+0000) 2019-10-18T00:57:29.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:29.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:29.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:29.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:30.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:30.155+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:30.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:30.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:30.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:30.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:31.641 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:31.635+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:31.638500+0000) 2019-10-18T00:57:31.642 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:31.635+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:31.639670+0000) 2019-10-18T00:57:31.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:31.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:31.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:31.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:32.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:32.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:32.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:32.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:33.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:57:33.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:33.681241+0000) 2019-10-18T00:57:33.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:33.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:33.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:33.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:34.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:34.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:34.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:34.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:35.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:57:35.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:35.093737+0000) 2019-10-18T00:57:35.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:35.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:35.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:35.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:35.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:35.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:36.141 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:36.135+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:36.138703+0000) 2019-10-18T00:57:36.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:36.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:36.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:36.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:37.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:37.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:37.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:37.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:38.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:38.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:38.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:38.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:39.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:57:39.684+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:39.690119+0000) 2019-10-18T00:57:39.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:39.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:39.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:39.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:40.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:40.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:40.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:40.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:40.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:40.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:41.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:41.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:41.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:41.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:42.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:42.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:42.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:42.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:43.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:57:43.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:43.681424+0000) 2019-10-18T00:57:43.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:43.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:43.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:43.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:44.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:44.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:45.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:57:45.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:45.093839+0000) 2019-10-18T00:57:45.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:45.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:45.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:45.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:45.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:45.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:46.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:46.135+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:46.138808+0000) 2019-10-18T00:57:46.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:46.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:46.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:46.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:47.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:47.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:47.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:47.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:48.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:48.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:48.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:48.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:49.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:57:49.687+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:49.690284+0000) 2019-10-18T00:57:49.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:49.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:49.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:49.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:50.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:50.163+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:50.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:50.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:50.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:50.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:51.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:51.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:51.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:51.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:52.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:52.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:52.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:52.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:53.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:57:53.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:53.681604+0000) 2019-10-18T00:57:53.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:53.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:53.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:53.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:54.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:54.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:54.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:54.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:55.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:57:55.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:55.093941+0000) 2019-10-18T00:57:55.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:57:55.163+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:57:55.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:55.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:55.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:55.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:56.141 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:57:56.135+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:56.138915+0000) 2019-10-18T00:57:56.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:56.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:56.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:56.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:57.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:57.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:57.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:57.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:59.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:59.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:57:59.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:57:59.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:57:59.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:57:59.687+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:57:59.690432+0000) 2019-10-18T00:58:00.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:00.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:00.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:00.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:00.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:00.163+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:01.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:01.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:01.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:01.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:02.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:02.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:02.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:02.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:03.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:02.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:03.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:02.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:03.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:58:03.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:03.681783+0000) 2019-10-18T00:58:03.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:03.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:03.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:03.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:04.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:04.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:04.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:04.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:05.096 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:58:05.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:05.094041+0000) 2019-10-18T00:58:05.171 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:05.162+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:05.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:05.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:05.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:05.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:06.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:58:06.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:06.139021+0000) 2019-10-18T00:58:06.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:06.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:06.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:06.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:07.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:07.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:07.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:07.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:08.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:08.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:08.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:08.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:09.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:09.687+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:09.690598+0000) 2019-10-18T00:58:09.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:09.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:09.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:09.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:10.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:10.166+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:10.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:10.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:10.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:10.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:11.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:11.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:11.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:11.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:12.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:12.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:12.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:13.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:58:13.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:13.681964+0000) 2019-10-18T00:58:13.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:13.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:13.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:13.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:14.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:14.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:14.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:14.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:15.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:58:15.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:15.094140+0000) 2019-10-18T00:58:15.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:15.166+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:15.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:15.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:15.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:15.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:16.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:58:16.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:16.139126+0000) 2019-10-18T00:58:16.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:16.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:16.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:17.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:17.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:17.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:17.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:18.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:18.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:18.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:18.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:19.693 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:19.687+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:19.690770+0000) 2019-10-18T00:58:19.694 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:19.687+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:19.691870+0000) 2019-10-18T00:58:19.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:19.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:19.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:19.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:20.174 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:20.166+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:20.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:20.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:20.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:20.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:21.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:21.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:21.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:21.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:22.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:22.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:22.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:22.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:23.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:58:23.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:23.682144+0000) 2019-10-18T00:58:23.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:23.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:23.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:23.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:24.194 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:24.187+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:24.191249+0000) 2019-10-18T00:58:24.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:24.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:24.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:24.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:25.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:58:25.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:25.094241+0000) 2019-10-18T00:58:25.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:25.166+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:25.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:25.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:25.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:25.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:26.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:58:26.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:26.139229+0000) 2019-10-18T00:58:26.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:26.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:26.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:26.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:27.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:27.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:27.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:27.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:28.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:28.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:28.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:28.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:29.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:29.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:29.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:29.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:30.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:30.166+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:30.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:30.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:30.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:30.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:31.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:31.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:31.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:31.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:32.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:32.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:33.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:58:33.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:33.682371+0000) 2019-10-18T00:58:33.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:33.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:33.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:33.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:34.197 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:34.187+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:34.191432+0000) 2019-10-18T00:58:34.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:34.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:34.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:34.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:35.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:58:35.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:35.094338+0000) 2019-10-18T00:58:35.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:35.170+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:35.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:35.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:35.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:35.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:36.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:58:36.133+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:36.139331+0000) 2019-10-18T00:58:36.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:36.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:36.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:36.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:37.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:37.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:37.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:37.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:38.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:38.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:38.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:38.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:39.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:39.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:39.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:39.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:40.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:40.169+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:40.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:40.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:40.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:40.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:41.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:41.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:41.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:41.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:42.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:42.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:42.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:42.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:43.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:58:43.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:43.682555+0000) 2019-10-18T00:58:43.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:43.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:43.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:43.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:44.194 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:44.187+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:44.191615+0000) 2019-10-18T00:58:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:44.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:44.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:45.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:58:45.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:45.094442+0000) 2019-10-18T00:58:45.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:45.169+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:45.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:45.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:45.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:45.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:46.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:58:46.133+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:46.139435+0000) 2019-10-18T00:58:46.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:46.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:46.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:46.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:47.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:47.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:47.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:47.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:48.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:48.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:48.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:48.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:49.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:49.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:49.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:49.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:50.179 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:50.169+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:50.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:50.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:50.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:50.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:51.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:51.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:51.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:51.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:52.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:52.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:52.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:52.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:53.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:58:53.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:53.682733+0000) 2019-10-18T00:58:53.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:53.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:53.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:53.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:54.194 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:58:54.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:54.191794+0000) 2019-10-18T00:58:54.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:54.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:54.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:54.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:55.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:58:55.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:55.094544+0000) 2019-10-18T00:58:55.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:58:55.169+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:58:55.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:55.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:55.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:55.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:56.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:58:56.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:58:56.139536+0000) 2019-10-18T00:58:56.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:56.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:56.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:56.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:57.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:57.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:57.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:57.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:58.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:58.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:58.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:58.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:58:59.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:59.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:58:59.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:58:59.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:00.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:00.177+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:00.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:00.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:00.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:00.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:01.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:01.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:01.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:01.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:02.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:02.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:02.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:02.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:03.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:59:03.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:03.682913+0000) 2019-10-18T00:59:03.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:03.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:03.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:03.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:04.194 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:59:04.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:04.191972+0000) 2019-10-18T00:59:04.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:04.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:04.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:04.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:05.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:59:05.093+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:05.094646+0000) 2019-10-18T00:59:05.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:05.177+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:06.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:06.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:06.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:06.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:06.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:59:06.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:06.139607+0000) 2019-10-18T00:59:06.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:06.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:06.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:06.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:07.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:07.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:07.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:07.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:08.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:08.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:08.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:08.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:09.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:09.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:09.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:09.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:10.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:10.177+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:11.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:11.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:11.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:11.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:11.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:11.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:11.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:11.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:13.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:13.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:13.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:59:13.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:13.683096+0000) 2019-10-18T00:59:14.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:14.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:14.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:14.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:14.195 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:59:14.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:14.192183+0000) 2019-10-18T00:59:15.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:15.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:15.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:15.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:15.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:59:15.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:15.094746+0000) 2019-10-18T00:59:15.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:15.176+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:16.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:16.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:16.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:16.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:16.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:59:16.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:16.139712+0000) 2019-10-18T00:59:16.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:16.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:16.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:16.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:17.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:17.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:17.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:17.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:18.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:18.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:18.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:18.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:20.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:20.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:20.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:20.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:20.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:20.180+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:21.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:21.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:21.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:21.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:22.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:22.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:22.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:22.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:23.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:23.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:23.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:23.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:23.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:59:23.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:23.683311+0000) 2019-10-18T00:59:24.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:24.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:24.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:24.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:24.195 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:59:24.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:24.192370+0000) 2019-10-18T00:59:25.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:25.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:25.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:25.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:25.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:59:25.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:25.094849+0000) 2019-10-18T00:59:25.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:25.180+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:26.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:26.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:26.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:26.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:26.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:59:26.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:26.139813+0000) 2019-10-18T00:59:27.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:27.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:27.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:27.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:28.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:28.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:28.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:28.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:29.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:29.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:29.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:29.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:30.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:30.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:30.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:30.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:30.186 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:30.180+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:31.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:31.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:31.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:31.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:32.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:32.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:32.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:32.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:33.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:33.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:33.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:33.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:33.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:59:33.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:33.683501+0000) 2019-10-18T00:59:34.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:34.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:34.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:34.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:34.195 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:59:34.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:34.192557+0000) 2019-10-18T00:59:35.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:35.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:35.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:35.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:35.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:59:35.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:35.094951+0000) 2019-10-18T00:59:35.186 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:35.180+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:36.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:36.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:36.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:36.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:36.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:59:36.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:36.139918+0000) 2019-10-18T00:59:37.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:37.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:37.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:37.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:38.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:38.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:38.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:38.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:39.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:39.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:39.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:39.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:40.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:40.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:40.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:40.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:40.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:40.180+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:41.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:41.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:41.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:41.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:42.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:42.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:42.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:42.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:43.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:43.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:43.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:43.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:43.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:59:43.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:43.683693+0000) 2019-10-18T00:59:44.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:44.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:44.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:44.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:44.195 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:59:44.186+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:44.192735+0000) 2019-10-18T00:59:45.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:59:45.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:45.095053+0000) 2019-10-18T00:59:45.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:45.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:45.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:45.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:45.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:45.184+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:46.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:46.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:46.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:46.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:46.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:59:46.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:46.140013+0000) 2019-10-18T00:59:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:47.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:47.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:48.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:48.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:48.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:48.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:49.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:49.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:49.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:49.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:50.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:50.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:50.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:50.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:50.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:50.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:51.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:51.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:51.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:51.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:52.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:52.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:52.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:52.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:53.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:53.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:53.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:53.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:53.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T00:59:53.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:53.683877+0000) 2019-10-18T00:59:54.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:54.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:54.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:54.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:54.195 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T00:59:54.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:54.192919+0000) 2019-10-18T00:59:55.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:55.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:55.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:55.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:55.097 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T00:59:55.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:55.095155+0000) 2019-10-18T00:59:55.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T00:59:55.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T00:59:56.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:56.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:56.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:56.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:56.142 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T00:59:56.135+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-17T23:59:56.140115+0000) 2019-10-18T00:59:57.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:57.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:57.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:57.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:58.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:58.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:58.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:58.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T00:59:59.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:59.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T00:59:59.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T00:59:59.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:00.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:00.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:00.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:00.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:00.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:00.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:01.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:01.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:01.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:01.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:02.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:02.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:02.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:02.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:03.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:03.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:03.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:03.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:03.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:00:03.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:03.684089+0000) 2019-10-18T01:00:04.195 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:00:04.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:04.193114+0000) 2019-10-18T01:00:04.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:04.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:04.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:04.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:05.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:00:05.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:05.095254+0000) 2019-10-18T01:00:05.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:05.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:05.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:05.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:05.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:05.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:06.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:00:06.135+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:06.140215+0000) 2019-10-18T01:00:06.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:06.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:06.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:06.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:07.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:07.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:07.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:07.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:08.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:08.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:08.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:08.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:09.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:09.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:09.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:09.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:10.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:10.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:10.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:10.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:10.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:10.187+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:11.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:11.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:11.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:11.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:12.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:12.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:12.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:12.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:13.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:13.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:13.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:13.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:13.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:00:13.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:13.684268+0000) 2019-10-18T01:00:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:14.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:14.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:14.196 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:00:14.189+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:14.193301+0000) 2019-10-18T01:00:15.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:00:15.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:15.095352+0000) 2019-10-18T01:00:15.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:15.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:15.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:15.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:15.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:15.187+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:16.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:00:16.135+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:16.140342+0000) 2019-10-18T01:00:16.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:16.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:16.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:16.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:17.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:17.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:17.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:17.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:18.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:18.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:18.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:18.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:19.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:19.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:19.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:19.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:20.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:20.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:20.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:20.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:20.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:20.187+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:21.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:21.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:21.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:21.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:22.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:22.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:22.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:22.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:23.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:23.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:23.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:23.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:23.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:00:23.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:23.684453+0000) 2019-10-18T01:00:24.196 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:00:24.189+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:24.193482+0000) 2019-10-18T01:00:24.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:24.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:24.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:24.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:25.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:00:25.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:25.095455+0000) 2019-10-18T01:00:25.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:25.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:25.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:25.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:25.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:25.186+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:26.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:00:26.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:26.140453+0000) 2019-10-18T01:00:26.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:26.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:26.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:26.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:27.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:27.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:27.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:27.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:28.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:28.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:28.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:28.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:29.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:29.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:29.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:29.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:30.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:30.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:30.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:30.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:30.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:30.190+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:31.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:31.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:31.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:31.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:32.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:32.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:32.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:32.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:33.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:33.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:33.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:33.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:33.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:00:33.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:33.684631+0000) 2019-10-18T01:00:34.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:34.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:34.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:34.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:34.196 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:00:34.189+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:34.193694+0000) 2019-10-18T01:00:35.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:35.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:35.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:35.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:35.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:00:35.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:35.095553+0000) 2019-10-18T01:00:35.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:35.190+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:36.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:36.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:36.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:36.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:36.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:00:36.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:36.140561+0000) 2019-10-18T01:00:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:37.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:37.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:38.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:38.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:38.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:38.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:39.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:39.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:39.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:39.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:40.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:40.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:40.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:40.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:40.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:40.190+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:41.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:41.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:41.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:41.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:42.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:42.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:42.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:42.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:43.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:43.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:43.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:43.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:43.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:00:43.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:43.684813+0000) 2019-10-18T01:00:44.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:44.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:44.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:44.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:44.196 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:00:44.189+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:44.193875+0000) 2019-10-18T01:00:45.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:00:45.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:45.095654+0000) 2019-10-18T01:00:45.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:45.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:45.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:45.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:45.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:45.190+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:46.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:46.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:46.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:46.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:46.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:00:46.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:46.140664+0000) 2019-10-18T01:00:47.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:47.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:47.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:47.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:47.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:47.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:47.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:47.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:48.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:48.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:48.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:48.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:49.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:49.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:50.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:49.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:50.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:50.194+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:51.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:51.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:51.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:51.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:52.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:52.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:52.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:52.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:53.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:53.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:53.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:53.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:53.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:00:53.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:53.685007+0000) 2019-10-18T01:00:54.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:54.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:54.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:54.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:54.200 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:00:54.189+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:54.194057+0000) 2019-10-18T01:00:55.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:55.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:55.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:55.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:55.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:00:55.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:55.095756+0000) 2019-10-18T01:00:55.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:00:55.194+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:00:56.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:56.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:56.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:56.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:56.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:00:56.134+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:00:56.140770+0000) 2019-10-18T01:00:57.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:57.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:57.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:57.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:58.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:58.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:58.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:58.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:00:59.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:59.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:00:59.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:00:59.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:00.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:00.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:00.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:00.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:00.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:00.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:01.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:01.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:01.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:01.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:02.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:01.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:02.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:01.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:02.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:02.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:02.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:02.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:03.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:01:03.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:03.685218+0000) 2019-10-18T01:01:03.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:03.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:03.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:03.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:04.196 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:01:04.193+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:04.194236+0000) 2019-10-18T01:01:04.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:04.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:04.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:04.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:05.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:01:05.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:05.095863+0000) 2019-10-18T01:01:05.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:05.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:05.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:05.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:05.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:06.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:01:06.133+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:06.140879+0000) 2019-10-18T01:01:06.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:06.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:06.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:06.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:07.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:07.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:07.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:07.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:08.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:08.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:08.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:08.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:09.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:09.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:09.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:09.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:10.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:10.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:10.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:10.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:10.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:10.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:11.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:11.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:11.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:11.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:12.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:12.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:12.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:12.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:13.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:01:13.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:13.685397+0000) 2019-10-18T01:01:13.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:13.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:13.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:13.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:14.197 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:01:14.193+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:14.194411+0000) 2019-10-18T01:01:14.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:14.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:14.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:14.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:15.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:01:15.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:15.095961+0000) 2019-10-18T01:01:15.204 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:15.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:15.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:15.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:15.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:15.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:16.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:01:16.133+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:16.140987+0000) 2019-10-18T01:01:16.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:16.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:16.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:16.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:17.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:17.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:17.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:17.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:18.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:18.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:18.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:18.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:19.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:19.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:19.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:19.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:20.204 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:20.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:20.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:20.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:20.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:20.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:21.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:21.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:21.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:21.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:22.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:22.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:22.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:22.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:23.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:01:23.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:23.685577+0000) 2019-10-18T01:01:23.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:23.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:23.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:23.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:24.199 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:01:24.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:24.194622+0000) 2019-10-18T01:01:24.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:24.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:24.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:24.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:25.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:01:25.093+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:25.096056+0000) 2019-10-18T01:01:25.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:25.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:25.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:25.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:25.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:25.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:26.143 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:01:26.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:26.141093+0000) 2019-10-18T01:01:26.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:26.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:26.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:26.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:27.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:27.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:27.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:27.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:28.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:28.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:28.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:28.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:29.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:29.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:29.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:29.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:30.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:30.201+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:30.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:30.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:30.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:30.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:31.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:31.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:31.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:31.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:32.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:32.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:32.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:32.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:33.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:01:33.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:33.685767+0000) 2019-10-18T01:01:33.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:33.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:33.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:33.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:34.197 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:01:34.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:34.194801+0000) 2019-10-18T01:01:34.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:34.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:34.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:34.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:35.098 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:01:35.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:35.096155+0000) 2019-10-18T01:01:35.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:35.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:35.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:35.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:35.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:35.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:36.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:01:36.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:36.141198+0000) 2019-10-18T01:01:36.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:36.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:36.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:36.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:37.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:37.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:37.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:37.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:38.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:38.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:38.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:38.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:39.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:39.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:39.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:39.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:40.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:40.204+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:40.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:40.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:40.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:40.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:41.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:41.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:41.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:41.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:42.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:42.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:42.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:42.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:43.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:01:43.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:43.685960+0000) 2019-10-18T01:01:43.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:43.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:43.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:43.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:44.197 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:01:44.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:44.194982+0000) 2019-10-18T01:01:44.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:44.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:44.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:44.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:45.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:01:45.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:45.096255+0000) 2019-10-18T01:01:45.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:45.204+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:45.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:45.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:45.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:45.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:46.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:01:46.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:46.141295+0000) 2019-10-18T01:01:46.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:46.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:46.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:46.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:47.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:47.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:47.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:47.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:48.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:48.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:48.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:48.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:49.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:49.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:49.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:49.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:50.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:50.204+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:50.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:50.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:50.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:50.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:51.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:51.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:51.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:51.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:52.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:52.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:52.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:52.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:53.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:01:53.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:53.686154+0000) 2019-10-18T01:01:53.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:53.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:53.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:53.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:54.197 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:01:54.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:54.195166+0000) 2019-10-18T01:01:54.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:54.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:54.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:54.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:55.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:01:55.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:55.096355+0000) 2019-10-18T01:01:55.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:01:55.204+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:01:56.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:56.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:56.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:56.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:56.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:01:56.136+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:01:56.141395+0000) 2019-10-18T01:01:56.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:56.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:56.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:56.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:58.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:58.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:58.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:58.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:01:59.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:59.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:01:59.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:01:59.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:00.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:00.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:00.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:00.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:00.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:00.204+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:01.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:01.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:01.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:01.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:02.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:02.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:02.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:02.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:03.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:03.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:03.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:03.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:03.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:02:03.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:03.686429+0000) 2019-10-18T01:02:04.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:04.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:04.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:04.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:04.198 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:02:04.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:04.195356+0000) 2019-10-18T01:02:05.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:02:05.092+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:05.096454+0000) 2019-10-18T01:02:05.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:05.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:05.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:05.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:05.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:05.208+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:06.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:02:06.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:06.141462+0000) 2019-10-18T01:02:06.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:06.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:06.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:06.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:07.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:07.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:07.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:07.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:08.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:08.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:08.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:08.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:09.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:09.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:10.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:10.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:10.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:10.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:10.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:10.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:11.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:11.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:11.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:11.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:12.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:12.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:12.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:12.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:13.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:13.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:13.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:13.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:13.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:02:13.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:13.686584+0000) 2019-10-18T01:02:14.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:14.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:14.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:14.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:14.198 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:02:14.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:14.195547+0000) 2019-10-18T01:02:15.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:15.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:15.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:15.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:15.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:02:15.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:15.096555+0000) 2019-10-18T01:02:15.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:15.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:16.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:16.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:16.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:16.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:16.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:02:16.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:16.141566+0000) 2019-10-18T01:02:17.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:17.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:17.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:17.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:18.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:18.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:18.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:18.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:19.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:19.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:19.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:19.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:20.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:20.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:20.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:20.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:20.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:20.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:21.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:21.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:21.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:21.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:22.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:22.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:22.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:22.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:23.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:23.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:23.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:23.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:23.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:02:23.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:23.686764+0000) 2019-10-18T01:02:24.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:24.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:24.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:24.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:24.198 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:02:24.192+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:24.195729+0000) 2019-10-18T01:02:25.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:02:25.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:25.096656+0000) 2019-10-18T01:02:25.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:25.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:25.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:25.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:25.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:25.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:26.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:02:26.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:26.141664+0000) 2019-10-18T01:02:26.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:26.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:26.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:26.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:27.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:27.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:27.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:27.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:28.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:28.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:28.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:28.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:29.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:29.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:29.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:29.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:30.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:30.211+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:30.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:30.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:30.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:30.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:31.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:31.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:31.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:31.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:32.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:32.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:32.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:32.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:33.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:33.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:33.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:33.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:33.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:02:33.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:33.686951+0000) 2019-10-18T01:02:34.198 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:02:34.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:34.195944+0000) 2019-10-18T01:02:34.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:34.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:34.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:34.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:35.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:02:35.091+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:35.096751+0000) 2019-10-18T01:02:35.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:35.211+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:35.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:35.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:35.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:35.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:36.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:02:36.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:36.141771+0000) 2019-10-18T01:02:36.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:36.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:36.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:36.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:37.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:37.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:37.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:37.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:38.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:38.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:38.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:38.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:39.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:39.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:39.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:39.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:40.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:40.211+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:40.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:40.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:40.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:40.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:41.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:41.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:41.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:41.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:42.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:42.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:42.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:42.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:43.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:43.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:43.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:43.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:43.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:02:43.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:43.687132+0000) 2019-10-18T01:02:44.198 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:02:44.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:44.196087+0000) 2019-10-18T01:02:44.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:44.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:44.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:44.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:45.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:02:45.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:45.096851+0000) 2019-10-18T01:02:45.219 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:45.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:45.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:45.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:45.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:45.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:46.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:02:46.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:46.141876+0000) 2019-10-18T01:02:46.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:46.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:46.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:46.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:47.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:47.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:47.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:47.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:48.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:48.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:48.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:48.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:49.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:49.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:49.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:49.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:50.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:50.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:50.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:50.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:51.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:51.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:51.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:51.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:52.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:52.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:52.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:52.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:53.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:53.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:53.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:53.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:53.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:02:53.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:53.687313+0000) 2019-10-18T01:02:54.198 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:02:54.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:54.196246+0000) 2019-10-18T01:02:54.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:54.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:54.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:54.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:55.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:02:55.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:55.096956+0000) 2019-10-18T01:02:55.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:02:55.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:02:55.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:55.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:55.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:55.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:56.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:02:56.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:02:56.141980+0000) 2019-10-18T01:02:56.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:56.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:56.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:56.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:57.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:57.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:57.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:57.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:58.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:58.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:58.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:58.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:02:59.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:59.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:02:59.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:02:59.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:00.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:00.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:00.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:00.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:00.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:00.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:01.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:01.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:01.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:01.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:02.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:02.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:02.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:02.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:03.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:03.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:03.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:03.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:03.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:03:03.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:03.687491+0000) 2019-10-18T01:03:04.199 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:03:04.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:04.196435+0000) 2019-10-18T01:03:04.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:04.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:04.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:04.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:05.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:03:05.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:05.097054+0000) 2019-10-18T01:03:05.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:05.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:05.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:05.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:05.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:05.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:06.144 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:03:06.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:06.142082+0000) 2019-10-18T01:03:06.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:06.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:06.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:06.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:07.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:07.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:07.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:07.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:08.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:08.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:08.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:08.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:09.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:09.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:09.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:09.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:10.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:10.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:10.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:10.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:10.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:10.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:11.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:11.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:11.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:11.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:12.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:12.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:12.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:12.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:13.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:13.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:13.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:13.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:13.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:03:13.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:13.687673+0000) 2019-10-18T01:03:14.199 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:03:14.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:14.196626+0000) 2019-10-18T01:03:14.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:14.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:14.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:14.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:15.099 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:03:15.090+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:15.097153+0000) 2019-10-18T01:03:15.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:15.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:15.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:15.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:15.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:15.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:16.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:03:16.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:16.142185+0000) 2019-10-18T01:03:16.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:16.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:16.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:16.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:17.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:17.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:17.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:17.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:18.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:18.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:18.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:18.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:19.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:19.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:19.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:19.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:20.225 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:20.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:20.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:20.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:21.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:21.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:21.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:21.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:22.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:22.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:22.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:22.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:23.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:23.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:23.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:23.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:23.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:03:23.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:23.687859+0000) 2019-10-18T01:03:24.199 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:03:24.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:24.196806+0000) 2019-10-18T01:03:24.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:24.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:24.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:24.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:25.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:03:25.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:25.097257+0000) 2019-10-18T01:03:25.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:25.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:25.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:25.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:25.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:25.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:26.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:03:26.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:26.142286+0000) 2019-10-18T01:03:26.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:26.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:26.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:26.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:27.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:27.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:27.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:27.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:28.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:28.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:28.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:28.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:29.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:29.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:29.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:29.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:30.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:30.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:30.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:30.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:30.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:30.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:31.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:31.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:31.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:31.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:32.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:32.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:32.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:32.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:33.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:03:33.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:33.688070+0000) 2019-10-18T01:03:33.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:33.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:33.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:33.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:34.199 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:03:34.191+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:34.196953+0000) 2019-10-18T01:03:34.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:34.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:34.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:34.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:35.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:03:35.089+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:35.097359+0000) 2019-10-18T01:03:35.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:35.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:35.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:35.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:35.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:35.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:36.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:03:36.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:36.142390+0000) 2019-10-18T01:03:36.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:36.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:36.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:36.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:37.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:37.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:37.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:37.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:38.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:38.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:38.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:38.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:39.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:39.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:39.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:39.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:40.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:40.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:40.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:40.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:40.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:40.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:41.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:41.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:41.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:41.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:42.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:42.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:42.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:42.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:43.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:03:43.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:43.688251+0000) 2019-10-18T01:03:43.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:43.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:43.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:43.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:44.199 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:03:44.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:44.197131+0000) 2019-10-18T01:03:44.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:44.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:44.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:44.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:45.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:03:45.097+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:45.097462+0000) 2019-10-18T01:03:45.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:45.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:45.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:45.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:45.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:45.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:46.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:03:46.141+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:46.142494+0000) 2019-10-18T01:03:46.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:46.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:46.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:46.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:47.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:47.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:47.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:47.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:48.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:48.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:48.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:48.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:49.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:49.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:49.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:49.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:50.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:50.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:50.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:50.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:50.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:50.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:51.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:51.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:51.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:51.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:52.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:52.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:52.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:52.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:53.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:03:53.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:53.688441+0000) 2019-10-18T01:03:53.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:53.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:53.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:53.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:54.202 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:03:54.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:54.197268+0000) 2019-10-18T01:03:54.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:54.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:54.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:54.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:55.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:03:55.097+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:55.097562+0000) 2019-10-18T01:03:55.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:03:55.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:03:55.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:55.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:55.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:55.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:56.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:03:56.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:03:56.142600+0000) 2019-10-18T01:03:56.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:56.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:56.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:56.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:57.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:57.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:57.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:57.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:58.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:58.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:58.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:58.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:03:59.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:59.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:03:59.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:03:59.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:00.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:00.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:00.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:00.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:00.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:00.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:01.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:01.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:01.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:01.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:02.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:02.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:02.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:02.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:03.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:03.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:03.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:03.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:03.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:04:03.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:03.688655+0000) 2019-10-18T01:04:04.200 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:04:04.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:04.197421+0000) 2019-10-18T01:04:04.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:04.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:04.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:04.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:05.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:04:05.096+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:05.097666+0000) 2019-10-18T01:04:05.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:05.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:05.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:05.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:05.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:05.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:06.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:04:06.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:06.142660+0000) 2019-10-18T01:04:06.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:06.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T00:53:41.727896+0000 (oldest deadline 2019-10-18T00:54:01.727896+0000) 2019-10-18T01:04:06.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:06.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:07.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:07.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:08.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:08.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:09.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:09.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:10.234 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:10.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:10.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:10.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:11.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:11.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:12.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:12.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:13.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:13.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:13.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:04:13.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:13.688871+0000) 2019-10-18T01:04:14.200 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:04:14.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:14.197603+0000) 2019-10-18T01:04:14.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:14.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:15.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:04:15.096+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:15.097768+0000) 2019-10-18T01:04:15.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:15.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:15.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:15.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:16.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:04:16.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:16.142767+0000) 2019-10-18T01:04:16.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:16.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:17.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:17.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:18.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:18.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:19.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:19.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:20.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:20.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:20.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:20.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:21.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:21.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:22.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:22.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:23.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:04:23.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:23.689048+0000) 2019-10-18T01:04:23.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:23.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:24.200 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:04:24.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:24.197797+0000) 2019-10-18T01:04:24.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:24.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:25.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:04:25.096+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:25.097868+0000) 2019-10-18T01:04:25.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:25.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:25.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:25.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:26.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:04:26.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:26.142876+0000) 2019-10-18T01:04:26.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:26.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:27.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:27.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:27.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:27.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:28.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:28.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:28.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:28.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:29.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:29.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:29.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:29.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:30.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:30.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:30.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:30.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:30.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:30.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:31.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:31.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:31.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:31.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:32.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:32.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:32.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:32.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:33.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:04:33.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:33.689231+0000) 2019-10-18T01:04:33.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:33.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:33.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:33.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:34.200 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:04:34.190+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:34.197992+0000) 2019-10-18T01:04:34.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:34.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:34.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:34.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:35.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:04:35.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:35.097964+0000) 2019-10-18T01:04:35.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:35.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:35.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:35.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:35.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:35.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:36.145 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:04:36.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:36.142988+0000) 2019-10-18T01:04:36.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:36.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:36.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:36.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:37.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:37.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:37.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:37.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:38.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:38.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:38.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:38.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:39.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:39.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:39.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:39.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:40.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:40.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:40.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:40.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:40.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:40.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:41.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:41.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:41.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:41.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:42.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:42.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:42.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:42.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:43.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:43.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:43.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:43.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:43.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:04:43.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:43.689421+0000) 2019-10-18T01:04:44.200 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:04:44.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:44.198170+0000) 2019-10-18T01:04:44.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:44.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:44.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:44.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:45.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:04:45.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:45.098063+0000) 2019-10-18T01:04:45.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:45.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:45.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:45.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:45.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:45.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:46.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:04:46.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:46.143097+0000) 2019-10-18T01:04:46.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:46.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:46.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:46.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:47.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:47.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:47.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:47.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:48.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:48.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:48.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:48.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:49.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:49.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:49.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:49.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:50.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:50.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:50.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:50.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:50.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:50.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:51.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:51.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:51.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:51.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:52.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:52.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:52.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:52.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:53.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:53.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:53.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:53.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:53.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:04:53.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:53.689602+0000) 2019-10-18T01:04:54.201 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:04:54.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:54.198346+0000) 2019-10-18T01:04:54.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:54.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:54.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:54.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:55.100 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:04:55.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:55.098164+0000) 2019-10-18T01:04:55.242 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:04:55.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:04:55.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:55.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:55.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:55.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:56.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:04:56.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:04:56.143213+0000) 2019-10-18T01:04:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:56.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:56.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:57.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:57.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:57.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:57.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:58.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:58.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:58.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:58.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:04:59.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:59.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:04:59.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:04:59.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:00.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:00.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:00.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:00.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:00.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:00.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:01.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:01.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:01.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:01.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:02.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:02.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:02.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:02.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:03.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:03.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:03.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:03.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:03.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:05:03.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:03.689783+0000) 2019-10-18T01:05:04.201 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:05:04.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:04.198527+0000) 2019-10-18T01:05:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:04.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:04.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:05.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:05:05.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:05.098269+0000) 2019-10-18T01:05:05.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:05.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:05.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:05.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:05.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:05.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:06.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:05:06.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:06.143328+0000) 2019-10-18T01:05:06.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:06.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:06.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:06.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:07.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:07.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:07.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:07.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:08.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:08.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:08.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:08.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:09.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:09.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:10.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:10.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:10.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:10.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:10.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:10.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:11.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:11.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:11.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:11.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:12.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:12.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:12.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:12.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:13.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:13.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:13.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:13.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:13.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:05:13.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:13.689964+0000) 2019-10-18T01:05:14.201 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:05:14.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:14.198704+0000) 2019-10-18T01:05:14.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:14.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:14.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:14.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:15.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:05:15.094+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:15.098370+0000) 2019-10-18T01:05:15.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:15.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:15.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:15.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:15.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:15.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:16.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:05:16.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:16.143439+0000) 2019-10-18T01:05:16.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:16.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:16.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:16.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:17.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:17.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:17.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:18.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:18.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:18.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:18.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:19.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:19.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:19.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:19.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:20.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:20.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:20.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:20.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:20.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:20.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:21.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:21.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:21.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:21.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:22.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:22.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:22.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:22.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:23.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:23.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:23.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:23.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:23.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:05:23.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:23.690151+0000) 2019-10-18T01:05:24.201 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:05:24.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:24.198883+0000) 2019-10-18T01:05:24.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:24.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:24.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:24.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:25.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:05:25.094+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:25.098471+0000) 2019-10-18T01:05:25.247 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:25.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:25.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:25.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:25.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:25.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:26.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:05:26.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:26.143545+0000) 2019-10-18T01:05:26.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:26.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:26.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:26.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:27.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:27.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:27.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:27.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:28.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:28.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:28.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:28.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:29.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:29.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:29.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:29.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:30.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:30.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:30.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:30.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:30.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:30.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:31.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:31.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:31.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:31.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:32.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:32.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:32.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:32.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:33.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:33.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:33.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:33.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:33.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:05:33.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:33.690370+0000) 2019-10-18T01:05:34.201 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:05:34.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:34.199059+0000) 2019-10-18T01:05:34.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:34.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:34.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:34.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:35.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:05:35.094+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:35.098576+0000) 2019-10-18T01:05:35.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:35.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:35.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:35.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:35.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:35.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:36.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:05:36.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:36.143655+0000) 2019-10-18T01:05:36.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:36.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:36.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:36.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:37.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:37.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:37.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:37.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:38.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:38.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:38.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:38.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:39.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:39.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:39.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:39.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:40.250 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:40.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:40.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:40.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:40.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:40.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:41.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:41.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:41.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:41.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:42.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:42.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:42.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:42.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:43.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:43.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:43.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:43.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:43.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:05:43.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:43.690551+0000) 2019-10-18T01:05:44.202 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:05:44.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:44.199243+0000) 2019-10-18T01:05:44.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:44.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:44.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:44.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:45.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:05:45.093+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:45.098680+0000) 2019-10-18T01:05:45.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:45.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:45.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:45.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:45.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:45.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:46.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:05:46.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:46.143764+0000) 2019-10-18T01:05:46.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:46.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:46.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:46.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:47.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:47.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:47.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:47.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:48.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:48.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:48.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:48.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:49.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:49.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:49.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:49.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:50.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:50.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:50.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:50.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:50.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:50.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:51.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:51.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:51.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:51.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:52.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:52.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:52.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:52.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:53.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:05:53.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:53.690735+0000) 2019-10-18T01:05:53.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:53.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:53.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:53.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:54.202 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:05:54.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:54.199421+0000) 2019-10-18T01:05:54.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:54.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:54.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:54.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:55.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:05:55.093+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:55.098785+0000) 2019-10-18T01:05:55.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:05:55.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:05:55.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:55.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:55.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:55.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:56.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:05:56.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:05:56.143880+0000) 2019-10-18T01:05:56.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:56.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:56.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:56.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:57.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:57.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:57.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:57.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:58.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:58.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:58.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:58.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:05:59.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:59.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:05:59.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:05:59.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:00.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:00.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:00.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:00.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:00.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:00.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:01.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:01.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:01.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:01.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:02.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:02.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:03.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:06:03.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:03.690951+0000) 2019-10-18T01:06:03.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:03.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:03.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:03.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:04.203 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:06:04.197+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:04.199600+0000) 2019-10-18T01:06:04.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:04.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:04.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:04.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:05.104 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:06:05.097+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:05.098886+0000) 2019-10-18T01:06:05.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:05.249+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:05.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:05.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:05.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:05.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:06.146 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:06:06.141+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:06.143985+0000) 2019-10-18T01:06:06.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:06.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:06.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:06.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:07.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:07.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:07.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:07.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:08.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:08.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:08.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:08.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:09.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:09.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:09.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:09.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:10.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:10.249+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:10.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:10.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:10.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:10.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:11.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:11.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:11.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:11.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:12.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:12.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:12.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:12.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:13.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:13.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:13.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:13.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:13.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:06:13.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:13.691134+0000) 2019-10-18T01:06:14.202 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:06:14.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:14.199780+0000) 2019-10-18T01:06:14.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:14.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:14.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:14.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:15.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:06:15.097+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:15.098983+0000) 2019-10-18T01:06:15.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:15.249+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:15.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:15.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:15.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:15.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:16.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:06:16.141+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:16.144092+0000) 2019-10-18T01:06:16.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:16.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:16.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:16.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:17.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:17.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:18.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:18.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:18.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:18.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:19.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:19.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:19.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:19.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:20.257 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:20.252+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:20.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:20.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:20.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:20.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:21.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:21.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:21.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:21.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:22.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:22.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:22.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:22.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:23.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:06:23.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:23.691316+0000) 2019-10-18T01:06:23.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:23.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:23.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:23.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:24.202 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:06:24.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:24.199970+0000) 2019-10-18T01:06:24.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:24.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:24.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:24.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:25.101 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:06:25.096+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:25.099081+0000) 2019-10-18T01:06:25.257 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:25.252+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:25.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:25.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:25.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:25.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:26.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:06:26.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:26.144186+0000) 2019-10-18T01:06:26.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:26.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:26.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:26.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:27.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:27.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:27.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:27.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:28.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:28.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:28.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:28.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:29.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:29.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:29.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:29.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:30.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:30.252+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:30.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:30.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:30.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:30.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:31.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:31.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:31.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:31.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:32.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:32.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:32.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:32.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:33.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:06:33.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:33.691499+0000) 2019-10-18T01:06:33.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:33.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:33.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:33.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:34.203 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:06:34.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:34.200150+0000) 2019-10-18T01:06:34.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:34.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:34.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:34.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:35.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:06:35.096+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:35.099183+0000) 2019-10-18T01:06:35.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:35.252+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:35.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:35.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:35.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:35.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:36.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:06:36.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:36.144284+0000) 2019-10-18T01:06:36.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:36.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:36.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:36.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:37.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:37.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:37.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:37.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:38.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:38.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:38.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:38.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:39.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:39.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:39.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:39.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:40.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:40.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:40.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:40.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:40.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:40.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:41.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:41.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:41.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:41.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:42.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:42.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:42.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:42.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:43.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:06:43.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:43.691699+0000) 2019-10-18T01:06:43.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:43.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:43.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:43.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:44.203 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:06:44.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:44.200326+0000) 2019-10-18T01:06:44.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:44.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:44.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:44.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:45.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:06:45.096+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:45.099290+0000) 2019-10-18T01:06:45.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:45.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:45.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:45.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:45.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:45.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:46.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:06:46.140+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:46.144389+0000) 2019-10-18T01:06:46.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:46.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:46.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:46.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:47.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:47.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:47.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:47.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:48.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:48.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:48.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:48.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:49.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:49.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:49.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:49.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:50.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:50.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:50.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:50.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:50.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:50.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:51.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:51.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:51.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:51.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:52.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:52.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:52.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:52.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:53.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:06:53.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:53.691878+0000) 2019-10-18T01:06:53.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:53.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:53.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:53.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:54.203 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:06:54.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:54.200471+0000) 2019-10-18T01:06:54.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:54.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:54.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:54.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:55.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:06:55.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:55.099393+0000) 2019-10-18T01:06:55.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:06:55.255+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:06:55.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:55.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:55.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:55.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:56.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:06:56.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:06:56.144497+0000) 2019-10-18T01:06:56.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:56.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:56.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:56.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:57.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:57.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:57.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:57.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:58.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:58.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:58.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:58.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:06:59.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:59.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:06:59.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:06:59.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:00.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:00.255+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:00.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:00.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:00.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:00.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:01.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:01.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:01.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:01.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:02.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:02.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:02.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:02.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:03.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:07:03.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:03.692058+0000) 2019-10-18T01:07:03.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:03.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:03.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:03.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:04.204 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:07:04.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:04.200651+0000) 2019-10-18T01:07:04.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:04.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:04.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:04.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:05.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:07:05.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:05.099494+0000) 2019-10-18T01:07:05.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:05.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:05.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:05.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:05.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:05.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:06.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:07:06.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:06.144599+0000) 2019-10-18T01:07:06.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:06.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:06.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:06.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:07.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:07.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:07.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:07.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:08.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:08.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:08.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:08.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:09.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:09.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:09.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:09.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:10.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:10.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:10.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:10.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:10.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:10.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:11.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:11.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:11.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:11.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:12.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:12.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:12.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:12.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:13.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:07:13.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:13.692242+0000) 2019-10-18T01:07:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:13.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:13.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:14.203 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:07:14.196+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:14.200825+0000) 2019-10-18T01:07:14.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:14.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:14.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:14.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:15.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:07:15.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:15.099595+0000) 2019-10-18T01:07:15.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:15.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:15.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:15.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:15.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:15.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:16.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:07:16.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:16.144724+0000) 2019-10-18T01:07:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:16.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:16.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:16.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:17.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:17.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:17.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:17.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:18.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:18.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:18.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:18.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:19.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:19.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:19.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:19.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:20.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:20.259+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:20.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:20.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:20.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:20.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:21.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:21.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:21.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:21.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:22.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:22.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:22.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:22.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:23.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:07:23.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:23.692422+0000) 2019-10-18T01:07:23.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:23.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:23.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:23.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:24.203 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:07:24.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:24.200967+0000) 2019-10-18T01:07:24.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:24.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:24.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:24.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:25.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:07:25.095+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:25.099699+0000) 2019-10-18T01:07:25.269 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:25.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:25.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:25.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:25.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:25.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:26.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:07:26.139+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:26.144837+0000) 2019-10-18T01:07:26.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:26.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:26.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:26.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:27.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:27.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:27.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:27.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:28.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:28.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:28.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:28.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:29.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:29.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:29.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:29.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:30.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:30.262+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:30.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:30.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:30.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:30.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:31.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:31.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:31.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:31.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:32.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:32.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:32.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:32.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:33.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:33.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:33.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:33.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:33.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:07:33.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:33.692601+0000) 2019-10-18T01:07:34.204 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:07:34.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:34.201147+0000) 2019-10-18T01:07:34.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:34.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:34.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:34.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:35.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:07:35.094+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:35.099799+0000) 2019-10-18T01:07:35.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:35.262+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:35.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:35.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:35.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:35.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:36.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:07:36.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:36.144942+0000) 2019-10-18T01:07:36.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:36.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:36.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:36.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:37.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:37.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:37.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:37.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:38.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:38.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:38.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:38.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:39.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:39.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:39.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:39.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:40.272 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:40.262+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:40.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:40.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:40.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:40.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:41.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:41.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:41.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:41.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:42.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:42.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:42.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:42.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:43.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:43.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:43.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:43.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:43.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:07:43.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:43.692780+0000) 2019-10-18T01:07:44.204 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:07:44.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:44.201328+0000) 2019-10-18T01:07:44.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:44.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:44.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:44.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:45.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:07:45.094+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:45.099897+0000) 2019-10-18T01:07:45.272 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:45.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:45.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:45.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:45.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:45.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:46.147 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:07:46.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:46.145046+0000) 2019-10-18T01:07:46.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:46.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:46.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:46.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:47.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:47.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:47.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:47.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:48.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:48.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:48.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:48.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:49.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:49.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:49.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:49.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:50.274 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:50.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:50.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:50.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:50.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:50.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:51.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:51.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:51.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:51.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:52.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:52.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:52.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:52.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:53.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:07:53.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:53.692964+0000) 2019-10-18T01:07:53.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:53.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:53.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:53.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:54.204 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:07:54.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:54.201497+0000) 2019-10-18T01:07:54.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:54.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:54.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:54.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:55.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:07:55.094+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:55.099998+0000) 2019-10-18T01:07:55.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:07:55.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:07:55.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:55.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:55.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:55.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:56.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:07:56.138+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:07:56.145147+0000) 2019-10-18T01:07:56.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:56.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:56.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:56.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:57.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:57.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:57.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:57.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:58.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:58.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:58.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:58.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:07:59.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:59.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:07:59.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:07:59.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:00.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:00.266+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:00.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:00.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:00.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:00.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:01.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:01.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:01.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:01.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:02.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:02.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:02.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:02.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:03.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:08:03.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:03.693156+0000) 2019-10-18T01:08:03.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:03.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:03.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:03.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:04.204 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:08:04.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:04.201727+0000) 2019-10-18T01:08:04.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:04.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:04.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:04.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:05.102 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:05.093+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:05.100090+0000) 2019-10-18T01:08:05.276 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:05.265+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:05.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:05.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:05.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:05.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:06.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:08:06.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:06.145252+0000) 2019-10-18T01:08:06.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:06.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:06.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:06.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:07.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:07.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:07.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:07.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:08.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:08.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:08.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:08.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:09.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:09.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:09.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:09.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:10.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:10.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:10.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:10.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:10.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:10.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:11.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:11.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:11.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:11.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:12.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:12.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:12.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:12.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:13.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:08:13.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:13.693297+0000) 2019-10-18T01:08:13.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:13.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:13.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:13.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:14.205 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:08:14.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:14.201912+0000) 2019-10-18T01:08:14.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:14.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:14.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:14.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:15.103 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:15.093+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:15.100188+0000) 2019-10-18T01:08:15.104 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:15.097+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:15.101838+0000) 2019-10-18T01:08:15.278 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:15.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:15.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:15.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:15.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:16.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:08:16.137+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:16.145359+0000) 2019-10-18T01:08:16.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:16.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:16.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:16.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:17.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:17.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:18.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:18.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:18.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:18.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:19.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:19.593+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:19.600386+0000) 2019-10-18T01:08:19.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:19.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:19.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:19.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:20.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:20.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:20.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:20.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:20.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:20.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:21.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:21.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:21.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:21.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:22.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:22.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:22.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:22.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:23.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:23.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:23.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:23.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:23.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:08:23.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:23.693485+0000) 2019-10-18T01:08:24.205 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:08:24.195+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:24.202091+0000) 2019-10-18T01:08:24.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:24.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:24.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:24.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:25.280 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:25.273+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:25.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:25.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:25.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:25.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:26.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:08:26.145+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:26.145475+0000) 2019-10-18T01:08:26.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:26.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:26.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:26.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:27.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:27.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:28.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:28.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:28.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:28.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:29.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:29.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:29.600480+0000) 2019-10-18T01:08:29.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:29.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:29.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:29.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:30.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:30.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:30.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:30.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:30.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:30.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:31.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:31.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:31.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:31.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:32.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:32.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:32.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:32.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:33.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:08:33.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:33.693667+0000) 2019-10-18T01:08:33.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:33.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:33.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:33.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:34.205 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:08:34.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:34.202217+0000) 2019-10-18T01:08:34.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:34.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:34.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:34.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:35.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:35.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:35.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:35.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:35.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:35.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:36.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:08:36.145+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:36.145573+0000) 2019-10-18T01:08:36.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:36.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:36.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:36.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:37.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:37.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:37.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:37.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:38.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:38.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:38.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:38.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:39.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:39.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:39.600573+0000) 2019-10-18T01:08:39.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:39.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:39.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:39.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:40.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:40.276+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:40.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:40.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:40.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:40.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:41.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:41.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:41.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:41.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:42.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:42.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:42.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:42.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:43.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:08:43.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:43.693855+0000) 2019-10-18T01:08:43.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:43.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:43.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:43.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:44.205 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:08:44.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:44.202367+0000) 2019-10-18T01:08:44.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:44.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:44.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:44.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:45.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:45.276+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:45.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:45.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:45.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:45.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:46.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:08:46.144+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:46.145674+0000) 2019-10-18T01:08:46.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:46.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:46.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:46.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:47.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:47.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:47.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:47.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:48.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:48.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:48.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:48.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:49.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:49.596+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:49.600667+0000) 2019-10-18T01:08:49.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:49.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:49.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:49.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:50.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:50.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:50.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:50.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:50.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:50.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:51.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:51.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:51.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:51.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:52.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:52.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:52.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:52.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:53.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:08:53.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:53.694040+0000) 2019-10-18T01:08:53.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:53.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:53.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:53.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:54.205 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:08:54.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:54.202551+0000) 2019-10-18T01:08:54.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:54.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:54.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:54.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:55.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:08:55.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:08:55.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:55.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:55.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:55.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:56.150 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:08:56.144+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:56.145778+0000) 2019-10-18T01:08:56.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:56.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:56.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:56.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:57.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:57.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:57.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:57.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:58.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:58.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:58.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:58.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:08:59.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:08:59.596+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:08:59.600762+0000) 2019-10-18T01:08:59.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:59.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:08:59.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:08:59.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:00.287 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:00.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:00.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:00.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:00.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:00.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:01.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:01.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:01.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:01.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:02.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:02.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:02.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:02.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:03.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:09:03.690+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:03.694228+0000) 2019-10-18T01:09:03.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:03.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:03.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:03.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:04.205 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:09:04.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:04.202708+0000) 2019-10-18T01:09:04.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:04.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:04.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:04.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:05.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:05.280+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:05.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:05.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:05.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:05.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:06.149 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:06.144+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:06.145883+0000) 2019-10-18T01:09:06.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:06.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:06.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:06.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:07.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:07.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:07.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:07.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:08.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:08.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:08.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:08.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:09.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:09:09.596+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:09.600861+0000) 2019-10-18T01:09:09.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:09.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:09.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:09.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:10.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:10.284+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:10.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:10.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:10.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:10.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:11.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:11.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:12.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:12.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:12.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:12.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:13.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:09:13.690+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:13.694420+0000) 2019-10-18T01:09:13.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:13.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:13.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:13.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:14.207 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:09:14.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:14.202897+0000) 2019-10-18T01:09:14.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:14.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:14.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:14.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:15.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:15.284+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:15.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:15.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:15.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:15.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:16.149 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:16.143+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:16.145993+0000) 2019-10-18T01:09:16.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:16.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:16.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:16.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:17.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:17.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:17.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:17.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:18.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:18.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:18.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:18.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:19.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:09:19.595+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:19.600961+0000) 2019-10-18T01:09:19.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:19.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:19.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:19.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:20.290 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:20.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:20.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:20.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:20.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:20.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:21.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:21.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:21.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:21.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:22.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:22.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:22.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:22.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:23.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:09:23.690+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:23.694623+0000) 2019-10-18T01:09:23.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:23.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:23.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:23.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:24.206 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:09:24.198+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:24.203107+0000) 2019-10-18T01:09:24.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:24.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:24.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:24.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:25.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:25.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:25.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:25.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:25.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:25.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:26.148 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:26.143+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:26.146095+0000) 2019-10-18T01:09:26.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:26.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:26.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:26.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:27.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:27.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:27.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:27.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:28.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:28.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:28.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:28.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:29.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:09:29.595+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:29.601058+0000) 2019-10-18T01:09:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:29.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:29.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:30.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:30.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:30.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:30.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:30.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:30.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:31.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:31.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:31.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:31.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:32.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:32.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:32.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:32.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:33.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:09:33.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:33.694805+0000) 2019-10-18T01:09:33.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:33.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:33.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:33.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:34.206 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:09:34.202+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:34.203294+0000) 2019-10-18T01:09:34.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:34.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:34.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:34.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:35.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:35.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:35.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:35.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:35.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:35.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:36.149 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:36.143+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:36.146194+0000) 2019-10-18T01:09:36.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:36.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:36.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:36.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:37.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:37.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:37.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:37.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:38.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:38.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:38.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:38.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:39.603 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:09:39.595+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:39.601160+0000) 2019-10-18T01:09:39.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:39.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:39.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:39.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:40.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:40.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:40.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:40.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:40.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:40.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:41.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:41.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:41.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:41.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:42.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:42.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:42.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:42.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:43.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:43.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:43.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:09:43.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:43.694988+0000) 2019-10-18T01:09:44.206 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:09:44.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:44.203458+0000) 2019-10-18T01:09:44.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:44.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:44.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:44.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:45.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:45.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:45.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:45.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:45.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:45.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:46.149 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:46.143+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:46.146298+0000) 2019-10-18T01:09:46.150 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:46.143+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:46.147405+0000) 2019-10-18T01:09:46.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:46.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:46.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:46.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:47.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:47.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:47.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:47.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:48.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:48.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:48.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:48.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:49.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:09:49.595+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:49.601260+0000) 2019-10-18T01:09:49.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:49.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:49.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:49.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:50.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:50.287+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:50.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:50.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:50.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:50.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:50.649 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:09:50.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:50.646493+0000) 2019-10-18T01:09:51.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:51.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:51.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:51.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:52.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:52.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:52.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:52.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:53.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:53.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:53.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:53.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:53.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:09:53.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:53.695169+0000) 2019-10-18T01:09:54.206 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:09:54.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:54.203667+0000) 2019-10-18T01:09:54.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:54.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:54.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:54.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:55.297 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:09:55.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:09:55.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:55.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:55.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:55.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:56.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:56.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:56.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:56.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:57.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:57.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:57.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:57.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:58.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:58.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:58.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:58.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:59.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:59.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:09:59.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:09:59.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:09:59.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:09:59.594+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:09:59.601412+0000) 2019-10-18T01:10:00.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:00.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:00.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:00.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:00.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:00.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:00.649 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:10:00.642+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:00.646599+0000) 2019-10-18T01:10:01.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:01.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:01.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:01.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:02.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:02.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:02.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:02.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:03.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:03.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:03.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:03.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:03.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:10:03.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:03.695357+0000) 2019-10-18T01:10:04.206 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:10:04.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:04.203855+0000) 2019-10-18T01:10:04.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:04.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:04.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:04.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:05.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:05.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:05.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:05.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:05.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:05.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:06.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:06.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:06.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:06.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:07.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:07.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:07.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:07.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:08.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:08.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:08.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:08.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:09.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:10:09.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:09.601536+0000) 2019-10-18T01:10:09.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:09.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:09.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:09.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:10.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:10.290+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:10.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:10.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:10.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:10.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:10.649 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:10:10.642+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:10.646709+0000) 2019-10-18T01:10:11.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:11.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:11.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:11.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:12.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:12.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:12.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:12.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:13.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:13.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:13.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:13.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:13.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:10:13.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:13.695537+0000) 2019-10-18T01:10:14.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:10:14.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:14.204046+0000) 2019-10-18T01:10:14.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:14.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:14.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:14.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:15.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:15.294+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:15.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:15.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:15.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:15.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:16.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:16.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:16.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:16.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:17.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:17.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:17.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:17.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:18.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:18.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:18.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:18.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:19.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:19.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:19.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:19.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:19.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:10:19.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:19.601636+0000) 2019-10-18T01:10:20.301 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:20.294+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:20.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:20.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:20.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:20.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:20.649 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:10:20.642+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:20.646823+0000) 2019-10-18T01:10:21.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:21.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:21.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:21.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:22.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:22.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:22.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:22.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:23.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:23.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:23.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:23.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:23.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:10:23.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:23.695675+0000) 2019-10-18T01:10:24.207 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:10:24.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:24.204253+0000) 2019-10-18T01:10:24.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:24.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:24.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:24.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:25.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:25.294+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:25.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:25.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:25.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:25.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:26.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:26.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:26.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:26.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:27.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:27.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:27.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:27.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:28.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:28.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:28.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:28.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:29.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:29.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:29.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:29.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:29.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:10:29.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:29.601876+0000) 2019-10-18T01:10:30.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:30.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:30.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:30.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:30.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:30.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:30.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:10:30.641+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:30.646929+0000) 2019-10-18T01:10:31.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:31.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:31.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:31.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:32.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:32.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:32.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:32.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:33.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:33.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:33.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:33.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:33.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:10:33.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:33.695845+0000) 2019-10-18T01:10:34.207 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:10:34.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:34.204434+0000) 2019-10-18T01:10:34.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:34.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:34.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:34.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:35.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:35.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:35.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:35.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:35.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:35.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:36.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:36.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:36.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:36.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:37.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:37.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:37.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:37.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:38.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:38.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:38.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:38.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:39.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:39.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:39.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:39.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:39.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:10:39.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:39.601983+0000) 2019-10-18T01:10:40.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:40.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:40.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:40.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:40.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:40.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:40.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:10:40.641+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:40.647029+0000) 2019-10-18T01:10:41.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:41.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:41.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:41.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:42.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:42.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:42.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:42.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:43.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:43.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:43.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:43.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:43.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:10:43.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:43.696007+0000) 2019-10-18T01:10:44.207 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:10:44.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:44.204579+0000) 2019-10-18T01:10:44.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:44.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:44.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:44.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:45.306 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:45.297+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:45.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:45.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:45.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:45.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:46.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:46.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:46.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:46.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:47.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:47.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:47.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:47.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:48.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:48.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:48.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:48.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:49.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:49.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:49.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:10:49.601+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:49.602084+0000) 2019-10-18T01:10:50.306 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:50.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:50.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:50.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:50.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:50.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:50.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:10:50.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:50.647135+0000) 2019-10-18T01:10:51.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:51.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:51.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:51.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:52.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:52.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:52.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:52.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:53.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:53.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:53.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:53.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:53.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:10:53.693+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:53.696188+0000) 2019-10-18T01:10:54.207 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:10:54.201+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:54.204762+0000) 2019-10-18T01:10:54.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:54.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:54.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:54.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:55.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:10:55.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:10:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:55.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:55.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:56.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:56.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:56.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:56.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:57.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:57.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:57.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:57.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:58.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:58.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:59.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:59.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:10:59.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:10:59.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:10:59.604 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:10:59.601+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:10:59.602184+0000) 2019-10-18T01:11:00.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:00.301+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:00.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:00.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:00.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:00.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:00.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:11:00.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:00.647247+0000) 2019-10-18T01:11:01.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:01.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:01.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:01.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:02.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:02.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:02.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:02.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:03.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:03.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:03.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:03.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:03.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:11:03.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:03.696371+0000) 2019-10-18T01:11:04.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:11:04.200+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:04.204973+0000) 2019-10-18T01:11:04.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:04.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:04.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:04.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:05.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:05.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:05.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:05.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:05.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:05.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:06.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:06.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:06.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:06.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:07.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:07.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:07.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:07.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:08.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:08.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:08.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:08.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:09.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:09.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:09.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:09.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:11:09.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:09.602286+0000) 2019-10-18T01:11:10.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:10.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:10.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:10.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:10.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:10.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:10.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:11:10.644+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:10.647355+0000) 2019-10-18T01:11:11.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:11.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:11.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:11.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:12.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:12.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:12.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:12.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:13.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:13.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:13.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:13.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:13.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:11:13.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:13.696511+0000) 2019-10-18T01:11:14.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:11:14.200+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:14.205191+0000) 2019-10-18T01:11:14.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:14.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:14.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:14.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:15.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:15.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:15.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:15.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:15.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:15.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:16.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:16.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:16.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:16.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:17.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:17.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:17.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:17.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:18.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:18.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:18.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:18.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:19.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:19.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:19.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:19.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:19.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:11:19.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:19.602386+0000) 2019-10-18T01:11:20.312 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:20.304+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:20.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:20.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:20.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:20.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:20.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:11:20.644+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:20.647459+0000) 2019-10-18T01:11:21.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:21.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:21.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:21.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:22.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:22.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:22.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:22.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:23.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:23.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:23.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:23.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:23.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:11:23.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:23.696687+0000) 2019-10-18T01:11:24.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:11:24.200+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:24.205401+0000) 2019-10-18T01:11:24.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:24.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:24.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:24.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:25.312 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:25.308+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:25.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:25.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:25.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:25.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:26.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:26.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:26.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:26.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:27.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:27.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:27.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:27.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:28.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:28.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:28.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:28.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:29.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:29.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:29.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:29.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:29.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:11:29.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:29.602490+0000) 2019-10-18T01:11:30.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:30.308+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:30.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:30.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:30.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:30.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:30.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:11:30.644+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:30.647568+0000) 2019-10-18T01:11:31.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:31.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:31.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:31.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:32.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:32.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:32.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:32.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:33.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:33.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:33.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:33.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:33.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:11:33.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:33.696873+0000) 2019-10-18T01:11:34.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:11:34.200+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:34.205585+0000) 2019-10-18T01:11:34.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:34.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:34.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:34.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:35.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:35.308+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:35.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:35.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:35.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:35.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:36.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:36.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:36.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:36.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:37.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:37.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:37.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:37.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:38.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:38.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:38.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:38.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:39.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:39.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:39.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:39.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:39.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:11:39.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:39.602594+0000) 2019-10-18T01:11:40.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:40.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:40.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:40.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:40.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:40.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:40.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:11:40.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:40.647649+0000) 2019-10-18T01:11:41.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:41.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:41.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:41.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:42.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:42.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:42.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:42.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:43.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:43.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:43.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:43.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:43.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:11:43.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:43.697060+0000) 2019-10-18T01:11:44.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:11:44.200+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:44.205767+0000) 2019-10-18T01:11:44.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:44.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:44.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:44.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:45.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:45.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:45.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:45.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:45.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:45.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:46.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:46.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:46.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:46.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:47.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:47.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:47.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:47.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:48.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:48.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:48.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:48.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:49.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:49.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:49.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:49.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:49.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:11:49.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:49.602696+0000) 2019-10-18T01:11:50.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:50.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:50.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:50.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:50.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:50.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:50.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:11:50.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:50.647752+0000) 2019-10-18T01:11:51.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:51.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:51.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:51.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:52.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:52.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:52.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:52.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:53.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:53.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:53.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:53.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:53.700 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:11:53.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:53.697241+0000) 2019-10-18T01:11:54.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:11:54.200+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:54.205955+0000) 2019-10-18T01:11:54.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:54.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:54.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:54.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:55.318 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:11:55.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:11:55.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:55.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:55.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:55.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:56.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:56.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:56.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:56.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:57.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:57.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:57.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:57.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:58.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:58.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:58.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:58.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:59.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:59.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:11:59.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:11:59.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:11:59.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:11:59.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:11:59.602794+0000) 2019-10-18T01:12:00.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:00.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:00.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:00.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:00.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:00.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:00.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:12:00.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:00.647849+0000) 2019-10-18T01:12:01.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:01.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:01.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:01.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:02.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:02.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:02.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:02.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:03.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:03.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:03.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:03.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:03.700 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:12:03.692+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:03.697424+0000) 2019-10-18T01:12:04.209 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:04.204+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:04.206166+0000) 2019-10-18T01:12:04.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:04.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:04.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:04.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:05.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:05.311+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:05.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:05.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:05.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:05.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:06.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:06.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:06.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:06.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:07.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:07.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:08.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:08.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:08.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:08.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:09.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:09.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:09.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:09.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:09.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:12:09.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:09.602896+0000) 2019-10-18T01:12:10.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:10.315+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:10.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:10.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:10.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:10.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:10.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:12:10.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:10.647953+0000) 2019-10-18T01:12:11.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:11.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:11.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:11.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:12.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:12.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:12.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:12.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:13.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:13.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:13.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:13.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:13.700 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:12:13.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:13.697607+0000) 2019-10-18T01:12:14.209 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:14.203+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:14.206328+0000) 2019-10-18T01:12:14.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:14.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:14.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:14.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:15.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:15.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:15.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:15.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:15.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:15.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:16.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:16.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:16.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:16.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:17.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:17.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:17.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:17.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:18.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:18.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:18.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:18.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:19.605 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:12:19.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:19.602996+0000) 2019-10-18T01:12:19.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:19.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:19.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:19.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:20.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:20.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:20.650 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:12:20.642+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:20.648059+0000) 2019-10-18T01:12:20.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:20.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:20.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:20.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:21.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:21.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:21.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:21.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:22.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:22.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:22.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:22.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:23.700 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:12:23.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:23.697791+0000) 2019-10-18T01:12:23.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:23.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:23.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:23.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:24.209 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:24.203+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:24.206479+0000) 2019-10-18T01:12:24.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:24.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:24.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:24.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:25.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:25.314+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:25.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:25.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:25.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:25.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:26.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:26.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:26.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:26.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:27.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:27.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:27.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:27.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:28.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:28.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:28.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:28.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:29.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:12:29.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:29.603093+0000) 2019-10-18T01:12:29.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:29.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:29.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:29.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:30.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:30.318+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:30.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:12:30.642+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:30.648173+0000) 2019-10-18T01:12:30.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:30.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:30.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:30.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:31.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:31.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:31.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:31.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:32.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:32.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:32.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:32.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:33.700 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:12:33.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:33.697979+0000) 2019-10-18T01:12:33.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:33.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:33.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:33.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:34.209 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:34.203+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:34.206686+0000) 2019-10-18T01:12:34.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:34.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:34.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:34.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:35.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:35.318+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:35.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:35.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:35.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:35.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:36.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:36.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:36.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:36.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:37.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:37.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:37.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:37.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:38.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:38.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:38.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:38.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:39.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:12:39.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:39.603196+0000) 2019-10-18T01:12:39.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:39.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:39.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:39.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:40.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:40.318+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:40.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:12:40.642+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:40.648295+0000) 2019-10-18T01:12:40.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:40.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:40.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:40.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:41.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:41.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:41.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:41.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:42.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:42.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:42.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:42.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:43.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:43.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:43.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:43.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:43.701 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:12:43.695+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:43.698163+0000) 2019-10-18T01:12:44.210 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:44.203+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:44.206885+0000) 2019-10-18T01:12:44.210 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:44.203+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:44.207656+0000) 2019-10-18T01:12:44.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:44.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:44.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:44.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:45.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:45.318+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:45.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:45.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:45.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:45.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:46.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:46.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:46.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:46.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:47.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:47.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:47.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:47.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:48.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:48.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:48.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:48.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:48.711 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:48.703+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:48.707283+0000) 2019-10-18T01:12:49.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:49.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:49.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:49.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:49.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:12:49.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:49.603297+0000) 2019-10-18T01:12:50.327 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:50.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:50.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:50.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:50.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:50.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:50.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:12:50.641+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:50.648403+0000) 2019-10-18T01:12:51.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:51.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:51.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:51.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:52.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:52.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:53.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:53.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:53.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:53.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:53.701 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:12:53.695+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:53.698388+0000) 2019-10-18T01:12:54.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:54.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:54.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:54.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:55.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:12:55.321+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:12:55.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:55.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:55.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:55.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:56.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:56.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:56.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:56.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:57.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:57.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:57.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:57.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:58.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:58.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:58.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:58.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:58.711 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:12:58.703+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:58.707470+0000) 2019-10-18T01:12:59.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:59.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:12:59.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:12:59.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:12:59.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:12:59.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:12:59.603395+0000) 2019-10-18T01:13:00.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:00.321+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:00.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:00.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:00.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:00.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:00.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:13:00.641+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:00.648502+0000) 2019-10-18T01:13:01.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:01.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:01.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:01.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:02.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:02.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:02.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:02.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:03.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:03.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:03.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:03.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:03.701 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:13:03.695+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:03.698567+0000) 2019-10-18T01:13:04.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:04.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:04.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:04.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:05.331 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:05.321+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:05.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:05.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:05.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:05.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:06.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:06.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:06.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:06.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:07.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:07.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:07.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:07.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:08.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:08.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:08.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:08.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:08.710 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:13:08.703+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:08.707684+0000) 2019-10-18T01:13:09.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:09.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:09.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:09.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:09.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:13:09.601+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:09.603497+0000) 2019-10-18T01:13:10.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:10.325+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:10.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:10.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:10.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:10.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:10.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:13:10.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:10.648603+0000) 2019-10-18T01:13:11.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:11.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:11.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:11.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:12.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:12.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:12.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:12.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:13.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:13.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:13.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:13.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:13.701 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:13:13.695+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:13.698749+0000) 2019-10-18T01:13:14.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:14.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:14.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:14.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:15.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:15.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:15.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:15.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:15.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:15.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:16.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:16.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:16.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:16.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:17.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:17.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:17.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:17.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:18.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:18.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:18.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:18.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:18.710 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:13:18.702+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:18.707901+0000) 2019-10-18T01:13:19.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:19.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:19.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:19.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:19.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:13:19.601+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:19.603598+0000) 2019-10-18T01:13:20.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:20.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:20.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:20.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:20.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:20.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:20.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:13:20.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:20.648709+0000) 2019-10-18T01:13:21.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:21.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:21.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:21.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:22.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:22.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:23.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:23.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:23.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:23.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:23.701 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:13:23.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:23.698926+0000) 2019-10-18T01:13:24.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:24.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:24.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:24.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:25.335 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:25.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:25.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:25.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:25.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:26.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:26.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:26.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:26.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:27.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:27.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:27.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:27.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:28.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:28.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:28.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:28.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:28.711 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:13:28.702+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:28.708078+0000) 2019-10-18T01:13:29.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:29.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:29.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:29.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:29.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:13:29.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:29.603697+0000) 2019-10-18T01:13:30.335 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:30.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:30.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:30.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:30.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:30.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:30.657 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:13:30.644+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:30.648812+0000) 2019-10-18T01:13:31.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:31.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:31.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:31.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:32.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:32.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:33.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:33.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:33.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:33.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:33.702 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:13:33.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:33.699108+0000) 2019-10-18T01:13:34.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:34.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:34.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:34.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:35.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:35.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:35.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:35.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:35.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:35.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:36.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:36.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:36.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:36.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:37.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:37.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:37.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:37.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:38.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:38.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:38.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:38.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:38.711 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:13:38.702+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:38.708289+0000) 2019-10-18T01:13:39.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:39.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:39.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:39.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:39.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:13:39.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:39.603797+0000) 2019-10-18T01:13:40.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:40.332+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:40.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:40.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:40.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:40.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:40.651 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:13:40.644+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:40.648924+0000) 2019-10-18T01:13:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:41.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:41.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:41.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:42.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:42.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:42.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:42.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:43.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:43.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:43.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:43.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:43.702 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:13:43.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:43.699292+0000) 2019-10-18T01:13:44.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:44.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:44.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:44.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:45.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:45.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:45.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:45.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:45.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:45.332+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:46.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:46.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:46.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:46.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:47.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:47.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:47.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:47.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:48.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:48.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:48.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:48.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:48.711 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:13:48.702+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:48.708498+0000) 2019-10-18T01:13:49.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:49.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:49.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:49.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:49.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:13:49.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:49.603895+0000) 2019-10-18T01:13:50.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:50.332+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:50.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:50.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:50.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:50.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:50.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:13:50.644+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:50.649038+0000) 2019-10-18T01:13:51.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:51.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:51.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:51.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:52.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:52.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:52.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:52.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:53.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:53.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:53.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:53.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:53.702 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:13:53.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:53.699475+0000) 2019-10-18T01:13:54.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:54.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:54.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:54.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:55.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:13:55.332+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:13:55.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:55.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:55.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:55.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:56.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:56.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:56.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:56.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:57.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:57.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:57.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:57.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:58.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:58.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:58.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:58.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:58.711 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:13:58.702+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:58.708713+0000) 2019-10-18T01:13:59.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:59.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:13:59.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:13:59.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:13:59.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:13:59.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:13:59.603995+0000) 2019-10-18T01:14:00.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:00.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:00.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:00.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:00.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:00.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:00.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:14:00.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:00.649138+0000) 2019-10-18T01:14:01.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:01.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:01.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:01.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:02.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:02.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:02.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:02.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:03.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:03.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:03.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:03.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:03.702 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:14:03.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:03.699665+0000) 2019-10-18T01:14:04.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:04.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:04.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:04.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:05.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:05.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:05.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:05.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:05.342 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:05.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:06.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:06.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:06.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:06.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:07.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:07.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:07.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:07.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:08.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:08.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:08.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:08.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:08.712 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:14:08.702+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:08.708925+0000) 2019-10-18T01:14:09.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:09.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:09.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:09.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:09.606 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:14:09.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:09.604092+0000) 2019-10-18T01:14:10.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:10.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:10.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:10.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:10.343 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:10.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:10.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:14:10.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:10.649251+0000) 2019-10-18T01:14:11.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:11.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:11.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:11.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:12.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:12.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:12.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:12.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:13.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:13.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:13.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:13.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:13.702 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:14:13.694+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:13.699811+0000) 2019-10-18T01:14:14.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:14.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:14.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:14.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:15.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:15.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:15.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:15.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:15.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:15.335+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:16.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:16.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:16.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:16.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:17.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:17.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:17.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:17.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:18.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:18.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:18.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:18.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:18.712 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:14:18.706+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:18.709143+0000) 2019-10-18T01:14:19.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:19.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:19.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:19.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:19.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:14:19.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:19.604191+0000) 2019-10-18T01:14:20.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:20.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:20.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:20.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:20.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:20.339+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:20.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:14:20.643+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:20.649377+0000) 2019-10-18T01:14:21.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:21.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:21.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:21.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:22.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:22.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:22.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:22.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:23.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:23.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:23.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:23.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:23.703 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:14:23.698+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:23.700002+0000) 2019-10-18T01:14:24.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:24.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:24.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:24.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:25.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:25.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:25.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:25.339+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:26.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:26.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:26.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:26.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:27.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:27.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:27.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:27.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:28.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:28.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:28.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:28.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:28.713 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:14:28.706+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:28.709336+0000) 2019-10-18T01:14:29.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:29.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:29.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:29.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:29.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:14:29.599+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:29.604290+0000) 2019-10-18T01:14:30.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:30.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:30.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:30.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:30.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:30.339+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:30.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:14:30.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:30.649485+0000) 2019-10-18T01:14:31.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:31.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:31.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:31.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:32.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:32.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:32.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:32.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:33.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:33.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:33.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:33.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:33.703 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:14:33.697+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:33.700145+0000) 2019-10-18T01:14:34.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:34.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:34.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:34.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:35.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:35.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:35.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:35.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:35.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:35.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:36.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:36.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:36.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:36.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:37.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:37.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:37.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:37.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:38.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:38.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:38.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:38.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:38.712 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:14:38.705+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:38.709530+0000) 2019-10-18T01:14:39.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:39.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:39.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:39.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:39.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:14:39.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:39.604389+0000) 2019-10-18T01:14:40.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:40.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:40.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:40.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:40.347 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:40.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:40.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:14:40.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:40.649590+0000) 2019-10-18T01:14:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:41.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:04:07.554615+0000 (oldest deadline 2019-10-18T01:04:27.554615+0000) 2019-10-18T01:14:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:41.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:42.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:42.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:43.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:43.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:43.703 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:14:43.697+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:43.700340+0000) 2019-10-18T01:14:44.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:44.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:45.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:45.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:45.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:45.342+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:46.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:46.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:47.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:47.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:48.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:48.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:48.712 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:14:48.705+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:48.709745+0000) 2019-10-18T01:14:49.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:49.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:49.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:14:49.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:49.604491+0000) 2019-10-18T01:14:50.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:50.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:50.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:50.342+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:50.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:14:50.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:50.649696+0000) 2019-10-18T01:14:51.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:51.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:52.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:52.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:53.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:53.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:53.704 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:14:53.697+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:53.700523+0000) 2019-10-18T01:14:54.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:54.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:55.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:55.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:55.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:14:55.342+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:14:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:56.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:57.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:58.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:58.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:58.712 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:14:58.705+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:58.709959+0000) 2019-10-18T01:14:59.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:14:59.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:14:59.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:14:59.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:14:59.604587+0000) 2019-10-18T01:15:00.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:00.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:00.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:00.342+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:00.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:15:00.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:00.649816+0000) 2019-10-18T01:15:01.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:01.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:02.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:02.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:03.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:03.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:03.703 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:03.697+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:03.700706+0000) 2019-10-18T01:15:04.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:04.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:05.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:05.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:05.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:05.346+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:06.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:06.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:07.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:07.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:07.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:07.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:08.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:08.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:08.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:08.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:08.713 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:15:08.705+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:08.710148+0000) 2019-10-18T01:15:09.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:09.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:09.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:09.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:09.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:15:09.598+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:09.604682+0000) 2019-10-18T01:15:10.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:10.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:10.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:10.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:10.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:10.346+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:10.652 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:15:10.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:10.649928+0000) 2019-10-18T01:15:11.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:11.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:11.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:11.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:12.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:12.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:12.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:12.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:13.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:13.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:13.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:13.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:13.703 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:13.697+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:13.700877+0000) 2019-10-18T01:15:14.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:14.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:14.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:14.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:15.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:15.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:15.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:15.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:15.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:15.345+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:16.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:16.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:16.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:16.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:17.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:17.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:17.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:17.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:18.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:18.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:18.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:18.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:18.713 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:15:18.709+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:18.710384+0000) 2019-10-18T01:15:19.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:19.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:19.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:19.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:19.608 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:15:19.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:19.604778+0000) 2019-10-18T01:15:20.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:20.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:20.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:20.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:20.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:20.345+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:20.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:15:20.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:20.650037+0000) 2019-10-18T01:15:21.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:21.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:21.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:21.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:22.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:22.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:22.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:22.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:23.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:23.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:23.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:23.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:23.704 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:23.697+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:23.701049+0000) 2019-10-18T01:15:23.704 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:23.697+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:23.701909+0000) 2019-10-18T01:15:24.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:24.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:24.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:24.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:25.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:25.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:25.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:25.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:25.356 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:25.345+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:26.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:26.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:26.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:26.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:27.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:27.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:27.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:27.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:28.204 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:28.197+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:28.201467+0000) 2019-10-18T01:15:28.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:28.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:28.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:28.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:28.713 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:15:28.709+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:28.710588+0000) 2019-10-18T01:15:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:29.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:29.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:29.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:15:29.597+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:29.604874+0000) 2019-10-18T01:15:30.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:30.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:30.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:30.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:30.356 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:30.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:30.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:15:30.649+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:30.650154+0000) 2019-10-18T01:15:31.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:31.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:31.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:31.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:32.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:32.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:32.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:32.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:33.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:33.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:33.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:33.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:34.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:34.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:34.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:34.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:35.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:35.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:35.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:35.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:35.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:35.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:36.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:36.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:36.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:36.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:37.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:37.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:37.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:37.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:38.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:38.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:38.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:38.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:38.204 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:38.197+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:38.201657+0000) 2019-10-18T01:15:38.713 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:15:38.709+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:38.710778+0000) 2019-10-18T01:15:39.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:39.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:39.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:39.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:39.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:15:39.601+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:39.604970+0000) 2019-10-18T01:15:40.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:40.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:40.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:40.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:40.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:40.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:40.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:15:40.649+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:40.650265+0000) 2019-10-18T01:15:41.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:41.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:41.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:41.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:42.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:42.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:42.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:42.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:43.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:43.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:43.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:43.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:44.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:44.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:44.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:44.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:45.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:45.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:45.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:45.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:45.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:45.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:46.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:46.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:46.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:46.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:47.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:47.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:47.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:47.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:48.204 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:48.196+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:48.201849+0000) 2019-10-18T01:15:48.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:48.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:48.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:48.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:48.714 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:15:48.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:48.710991+0000) 2019-10-18T01:15:49.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:49.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:49.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:49.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:49.607 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:15:49.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:49.605064+0000) 2019-10-18T01:15:50.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:50.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:50.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:50.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:50.360 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:50.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:50.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:15:50.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:50.650373+0000) 2019-10-18T01:15:51.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:51.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:51.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:51.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:52.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:52.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:52.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:52.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:53.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:53.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:53.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:53.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:54.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:54.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:54.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:54.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:55.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:15:55.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:15:55.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:55.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:55.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:55.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:56.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:56.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:56.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:56.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:57.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:57.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:57.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:57.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:58.204 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:15:58.196+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:58.202031+0000) 2019-10-18T01:15:58.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:58.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:58.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:58.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:58.725 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:15:58.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:58.711207+0000) 2019-10-18T01:15:59.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:59.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:15:59.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:15:59.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:15:59.608 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:15:59.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:15:59.605165+0000) 2019-10-18T01:16:00.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:00.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:00.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:00.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:00.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:00.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:00.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:16:00.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:00.650482+0000) 2019-10-18T01:16:01.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:01.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:01.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:01.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:02.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:02.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:02.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:02.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:03.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:03.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:03.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:03.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:04.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:04.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:04.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:04.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:05.369 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:05.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:05.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:05.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:05.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:05.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:06.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:06.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:06.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:06.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:07.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:07.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:07.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:07.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:08.205 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:16:08.200+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:08.202224+0000) 2019-10-18T01:16:08.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:08.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:08.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:08.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:08.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:16:08.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:08.711417+0000) 2019-10-18T01:16:09.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:09.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:09.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:09.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:09.608 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:09.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:09.605268+0000) 2019-10-18T01:16:10.364 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:10.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:10.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:10.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:10.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:10.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:10.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:16:10.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:10.650592+0000) 2019-10-18T01:16:11.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:11.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:11.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:11.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:12.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:12.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:12.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:12.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:13.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:13.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:13.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:13.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:14.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:14.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:14.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:14.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:15.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:15.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:15.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:15.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:15.364 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:15.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:16.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:16.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:16.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:16.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:17.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:17.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:17.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:17.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:18.205 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:16:18.200+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:18.202409+0000) 2019-10-18T01:16:18.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:18.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:18.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:18.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:18.714 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:16:18.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:18.711630+0000) 2019-10-18T01:16:19.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:19.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:19.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:19.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:19.608 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:19.600+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:19.605360+0000) 2019-10-18T01:16:20.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:20.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:20.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:20.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:20.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:20.360+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:20.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:16:20.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:20.650697+0000) 2019-10-18T01:16:21.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:21.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:21.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:21.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:22.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:22.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:22.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:22.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:23.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:23.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:24.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:24.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:24.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:24.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:25.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:25.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:25.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:25.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:25.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:25.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:26.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:26.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:26.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:26.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:27.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:27.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:27.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:27.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:28.205 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:16:28.200+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:28.202590+0000) 2019-10-18T01:16:28.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:28.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:28.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:28.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:28.714 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:16:28.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:28.711838+0000) 2019-10-18T01:16:29.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:29.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:29.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:29.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:29.608 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:29.603+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:29.605471+0000) 2019-10-18T01:16:29.609 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:29.603+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:29.606984+0000) 2019-10-18T01:16:30.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:30.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:30.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:30.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:30.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:30.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:30.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:16:30.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:30.650807+0000) 2019-10-18T01:16:31.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:31.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:31.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:31.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:32.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:32.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:32.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:32.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:33.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:33.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:33.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:33.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:34.108 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:34.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:34.105659+0000) 2019-10-18T01:16:34.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:34.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:34.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:34.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:35.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:35.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:35.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:35.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:35.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:35.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:36.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:36.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:36.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:36.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:37.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:37.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:37.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:37.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:38.205 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:16:38.200+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:38.202814+0000) 2019-10-18T01:16:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:38.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:38.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:38.714 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:16:38.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:38.712024+0000) 2019-10-18T01:16:39.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:39.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:39.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:39.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:40.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:40.363+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:40.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:40.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:40.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:40.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:40.660 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:16:40.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:40.650929+0000) 2019-10-18T01:16:41.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:41.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:41.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:41.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:42.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:42.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:42.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:42.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:43.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:43.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:43.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:43.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:44.108 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:44.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:44.105752+0000) 2019-10-18T01:16:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:44.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:44.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:45.369 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:45.363+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:45.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:45.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:45.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:45.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:46.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:46.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:46.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:46.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:47.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:47.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:47.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:47.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:48.205 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:16:48.200+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:48.202997+0000) 2019-10-18T01:16:48.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:48.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:48.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:48.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:48.715 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:16:48.708+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:48.712204+0000) 2019-10-18T01:16:49.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:49.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:49.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:49.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:50.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:50.363+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:50.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:50.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:50.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:50.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:50.653 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:16:50.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:50.651039+0000) 2019-10-18T01:16:51.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:51.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:51.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:51.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:52.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:52.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:52.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:52.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:53.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:53.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:53.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:53.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:54.108 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:16:54.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:54.105846+0000) 2019-10-18T01:16:54.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:54.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:54.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:54.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:55.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:16:55.363+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:16:55.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:55.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:55.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:55.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:56.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:56.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:56.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:56.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:57.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:57.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:57.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:57.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:58.206 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:16:58.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:58.203178+0000) 2019-10-18T01:16:58.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:58.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:58.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:58.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:16:58.715 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:16:58.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:16:58.712346+0000) 2019-10-18T01:16:59.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:59.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:16:59.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:16:59.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:00.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:00.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:00.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:00.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:00.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:00.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:00.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:17:00.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:00.651146+0000) 2019-10-18T01:17:01.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:01.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:01.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:01.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:02.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:02.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:02.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:02.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:03.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:03.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:03.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:03.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:04.108 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:17:04.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:04.105943+0000) 2019-10-18T01:17:04.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:04.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:04.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:04.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:05.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:05.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:05.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:05.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:05.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:05.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:06.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:06.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:06.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:06.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:07.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:07.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:07.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:07.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:08.206 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:17:08.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:08.203368+0000) 2019-10-18T01:17:08.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:08.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:08.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:08.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:08.715 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:17:08.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:08.712550+0000) 2019-10-18T01:17:09.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:09.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:09.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:09.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:10.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:10.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:10.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:10.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:10.374 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:10.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:10.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:17:10.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:10.651249+0000) 2019-10-18T01:17:11.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:11.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:11.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:11.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:12.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:12.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:12.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:12.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:13.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:13.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:13.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:13.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:14.108 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:17:14.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:14.106041+0000) 2019-10-18T01:17:14.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:14.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:14.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:14.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:15.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:15.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:15.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:15.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:15.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:15.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:16.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:16.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:16.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:16.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:17.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:17.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:17.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:17.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:18.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:17:18.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:18.203547+0000) 2019-10-18T01:17:18.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:18.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:18.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:18.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:18.715 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:17:18.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:18.712761+0000) 2019-10-18T01:17:19.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:19.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:19.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:19.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:20.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:20.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:20.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:20.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:20.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:20.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:17:20.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:20.651351+0000) 2019-10-18T01:17:21.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:21.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:21.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:21.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:22.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:22.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:22.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:22.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:23.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:23.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:23.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:23.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:24.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:17:24.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:24.106140+0000) 2019-10-18T01:17:24.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:24.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:24.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:24.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:25.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:25.370+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:25.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:25.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:25.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:25.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:26.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:26.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:26.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:26.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:27.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:27.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:27.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:27.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:28.206 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:17:28.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:28.203731+0000) 2019-10-18T01:17:28.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:28.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:28.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:28.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:28.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:17:28.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:28.712937+0000) 2019-10-18T01:17:29.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:29.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:29.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:29.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:30.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:30.370+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:30.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:30.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:30.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:30.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:30.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:17:30.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:30.651455+0000) 2019-10-18T01:17:31.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:31.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:31.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:31.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:32.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:32.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:32.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:32.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:33.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:33.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:33.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:33.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:34.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:17:34.101+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:34.106243+0000) 2019-10-18T01:17:34.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:34.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:34.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:34.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:35.378 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:35.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:35.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:35.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:35.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:35.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:36.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:36.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:36.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:36.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:37.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:37.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:37.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:37.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:38.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:17:38.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:38.203914+0000) 2019-10-18T01:17:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:38.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:38.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:38.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:17:38.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:38.713148+0000) 2019-10-18T01:17:39.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:39.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:39.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:39.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:40.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:40.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:40.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:40.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:40.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:40.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:40.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:17:40.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:40.651561+0000) 2019-10-18T01:17:41.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:41.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:41.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:41.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:42.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:42.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:42.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:42.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:43.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:43.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:43.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:43.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:44.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:17:44.101+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:44.106343+0000) 2019-10-18T01:17:44.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:44.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:44.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:44.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:45.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:45.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:45.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:45.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:45.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:45.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:46.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:46.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:46.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:46.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:47.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:47.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:47.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:47.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:48.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:17:48.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:48.204125+0000) 2019-10-18T01:17:48.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:48.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:48.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:48.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:48.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:17:48.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:48.713329+0000) 2019-10-18T01:17:49.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:49.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:49.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:49.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:50.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:50.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:50.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:50.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:50.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:50.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:50.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:17:50.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:50.651674+0000) 2019-10-18T01:17:51.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:51.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:51.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:51.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:52.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:52.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:52.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:52.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:53.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:53.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:53.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:53.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:54.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:17:54.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:54.106439+0000) 2019-10-18T01:17:54.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:54.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:54.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:54.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:55.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:55.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:55.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:55.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:55.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:17:55.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:17:56.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:56.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:56.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:56.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:57.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:57.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:57.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:57.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:58.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:17:58.199+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:58.204306+0000) 2019-10-18T01:17:58.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:58.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:58.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:58.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:17:58.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:17:58.707+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:17:58.713522+0000) 2019-10-18T01:17:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:59.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:17:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:17:59.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:00.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:00.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:00.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:00.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:00.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:00.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:18:00.649+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:00.651783+0000) 2019-10-18T01:18:01.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:01.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:01.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:01.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:02.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:02.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:02.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:02.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:03.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:03.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:03.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:03.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:04.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:18:04.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:04.106535+0000) 2019-10-18T01:18:04.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:04.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:04.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:04.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:05.384 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:05.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:05.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:05.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:05.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:05.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:06.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:06.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:06.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:06.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:07.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:07.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:07.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:07.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:08.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:18:08.198+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:08.204494+0000) 2019-10-18T01:18:08.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:08.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:08.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:08.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:08.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:18:08.706+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:08.713736+0000) 2019-10-18T01:18:09.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:09.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:09.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:09.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:10.384 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:10.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:10.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:10.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:10.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:10.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:10.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:18:10.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:10.651888+0000) 2019-10-18T01:18:11.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:11.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:11.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:11.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:12.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:12.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:12.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:12.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:13.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:13.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:13.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:13.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:14.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:18:14.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:14.106597+0000) 2019-10-18T01:18:14.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:14.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:14.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:14.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:15.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:15.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:15.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:15.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:15.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:15.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:16.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:16.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:16.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:16.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:17.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:17.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:17.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:17.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:18.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:18:18.198+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:18.204707+0000) 2019-10-18T01:18:18.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:18.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:18.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:18.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:18.716 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:18:18.706+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:18.713947+0000) 2019-10-18T01:18:19.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:19.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:19.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:19.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:20.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:20.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:20.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:20.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:20.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:20.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:20.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:18:20.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:20.651995+0000) 2019-10-18T01:18:21.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:21.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:21.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:21.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:22.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:22.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:22.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:22.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:23.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:23.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:23.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:23.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:24.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:18:24.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:24.106697+0000) 2019-10-18T01:18:24.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:24.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:24.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:24.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:25.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:25.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:25.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:25.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:25.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:25.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:26.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:26.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:26.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:26.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:27.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:27.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:27.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:27.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:28.207 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:18:28.198+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:28.204890+0000) 2019-10-18T01:18:28.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:28.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:28.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:28.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:28.717 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:18:28.710+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:28.714160+0000) 2019-10-18T01:18:29.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:29.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:29.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:29.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:30.388 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:30.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:30.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:30.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:30.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:30.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:30.654 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:18:30.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:30.652099+0000) 2019-10-18T01:18:31.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:31.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:31.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:31.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:32.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:32.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:32.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:32.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:33.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:33.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:33.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:33.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:34.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:18:34.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:34.106798+0000) 2019-10-18T01:18:34.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:34.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:34.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:34.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:35.389 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:35.384+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:35.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:35.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:35.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:35.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:36.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:36.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:36.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:36.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:37.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:37.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:37.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:37.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:38.210 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:18:38.198+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:38.205073+0000) 2019-10-18T01:18:38.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:38.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:38.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:38.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:38.717 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:18:38.710+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:38.714389+0000) 2019-10-18T01:18:39.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:39.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:39.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:39.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:40.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:40.384+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:40.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:40.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:40.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:40.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:40.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:18:40.648+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:40.652201+0000) 2019-10-18T01:18:41.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:41.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:41.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:41.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:42.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:42.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:42.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:42.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:43.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:43.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:43.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:43.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:44.109 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:18:44.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:44.106895+0000) 2019-10-18T01:18:44.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:44.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:44.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:44.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:45.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:45.383+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:45.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:45.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:45.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:45.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:46.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:46.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:46.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:46.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:47.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:47.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:47.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:47.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:48.208 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:18:48.198+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:48.205219+0000) 2019-10-18T01:18:48.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:48.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:48.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:48.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:48.717 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:18:48.710+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:48.714586+0000) 2019-10-18T01:18:49.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:49.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:49.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:49.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:50.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:50.383+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:50.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:50.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:50.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:50.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:50.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:18:50.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:50.652312+0000) 2019-10-18T01:18:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:51.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:51.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:52.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:52.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:52.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:52.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:53.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:53.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:53.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:53.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:54.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:18:54.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:54.106991+0000) 2019-10-18T01:18:54.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:54.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:54.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:54.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:55.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:18:55.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:18:55.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:55.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:55.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:55.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:56.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:56.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:56.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:56.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:57.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:57.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:57.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:57.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:58.208 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:18:58.198+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:58.205411+0000) 2019-10-18T01:18:58.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:58.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:58.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:58.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:18:58.717 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:18:58.710+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:18:58.714760+0000) 2019-10-18T01:18:59.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:59.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:18:59.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:18:59.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:00.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:00.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:00.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:00.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:00.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:00.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:00.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:19:00.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:00.652424+0000) 2019-10-18T01:19:01.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:01.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:01.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:01.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:02.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:02.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:02.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:02.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:03.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:03.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:03.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:03.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:04.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:19:04.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:04.107095+0000) 2019-10-18T01:19:04.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:04.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:04.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:04.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:05.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:05.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:05.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:05.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:06.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:06.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:06.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:06.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:07.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:07.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:07.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:07.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:08.209 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:19:08.202+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:08.205628+0000) 2019-10-18T01:19:08.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:08.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:08.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:08.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:08.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:19:08.714+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:08.714973+0000) 2019-10-18T01:19:09.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:09.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:09.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:09.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:10.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:10.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:10.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:10.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:10.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:10.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:10.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:19:10.647+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:10.652530+0000) 2019-10-18T01:19:11.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:11.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:11.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:11.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:12.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:12.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:12.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:12.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:13.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:13.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:13.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:13.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:14.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:19:14.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:14.107196+0000) 2019-10-18T01:19:14.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:14.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:14.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:14.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:15.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:15.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:15.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:15.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:16.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:16.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:16.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:16.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:17.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:17.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:17.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:17.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:18.208 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:19:18.201+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:18.205814+0000) 2019-10-18T01:19:18.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:18.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:18.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:18.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:18.718 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:19:18.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:18.715184+0000) 2019-10-18T01:19:19.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:19.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:19.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:19.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:20.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:20.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:20.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:20.390+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:20.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:19:20.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:20.652640+0000) 2019-10-18T01:19:21.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:21.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:21.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:21.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:22.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:22.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:22.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:22.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:23.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:23.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:23.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:23.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:24.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:19:24.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:24.107293+0000) 2019-10-18T01:19:24.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:24.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:24.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:24.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:25.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:25.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:25.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:25.390+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:26.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:26.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:26.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:26.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:27.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:27.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:27.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:27.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:28.209 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:19:28.201+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:28.205994+0000) 2019-10-18T01:19:28.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:28.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:28.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:28.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:28.718 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:19:28.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:28.715400+0000) 2019-10-18T01:19:29.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:29.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:29.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:29.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:30.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:30.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:30.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:30.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:30.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:30.390+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:30.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:19:30.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:30.652745+0000) 2019-10-18T01:19:31.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:31.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:31.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:31.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:32.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:32.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:32.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:32.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:33.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:33.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:33.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:33.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:34.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:19:34.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:34.107392+0000) 2019-10-18T01:19:34.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:34.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:34.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:34.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:35.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:35.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:35.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:35.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:35.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:35.390+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:36.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:36.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:36.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:36.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:37.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:37.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:37.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:37.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:38.209 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:19:38.205+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:38.206198+0000) 2019-10-18T01:19:38.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:38.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:38.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:38.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:38.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:19:38.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:38.715607+0000) 2019-10-18T01:19:39.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:39.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:39.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:39.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:40.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:40.390+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:40.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:40.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:40.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:40.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:40.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:19:40.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:40.652843+0000) 2019-10-18T01:19:41.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:41.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:41.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:41.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:42.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:42.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:42.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:42.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:43.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:43.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:43.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:43.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:44.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:19:44.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:44.107492+0000) 2019-10-18T01:19:44.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:44.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:44.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:44.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:45.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:45.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:45.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:45.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:45.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:45.394+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:46.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:46.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:46.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:46.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:47.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:47.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:47.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:47.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:48.209 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:19:48.205+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:48.206382+0000) 2019-10-18T01:19:48.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:48.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:48.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:48.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:48.718 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:19:48.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:48.715786+0000) 2019-10-18T01:19:49.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:49.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:49.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:49.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:50.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:50.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:50.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:50.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:50.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:50.394+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:50.655 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:19:50.646+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:50.652954+0000) 2019-10-18T01:19:51.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:51.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:51.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:51.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:52.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:52.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:52.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:52.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:53.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:53.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:53.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:53.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:54.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:19:54.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:54.107592+0000) 2019-10-18T01:19:54.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:54.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:54.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:54.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:55.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:55.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:55.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:55.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:55.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:19:55.393+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:19:56.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:56.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:56.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:56.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:57.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:57.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:57.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:57.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:58.211 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:19:58.205+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:58.206592+0000) 2019-10-18T01:19:58.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:58.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:58.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:58.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:19:58.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:19:58.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:19:58.715997+0000) 2019-10-18T01:19:59.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:59.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:19:59.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:19:59.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:00.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:00.393+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:00.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:00.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:00.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:00.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:00.656 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:00.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:00.653069+0000) 2019-10-18T01:20:01.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:01.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:01.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:01.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:02.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:02.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:02.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:02.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:03.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:03.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:03.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:03.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:04.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:20:04.101+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:04.107700+0000) 2019-10-18T01:20:04.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:04.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:04.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:04.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:05.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:05.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:05.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:05.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:05.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:05.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:06.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:06.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:07.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:07.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:07.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:07.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:08.209 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:20:08.205+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:08.206780+0000) 2019-10-18T01:20:08.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:08.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:08.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:08.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:08.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:08.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:08.716183+0000) 2019-10-18T01:20:09.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:09.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:09.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:09.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:10.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:10.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:10.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:10.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:10.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:10.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:10.656 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:10.645+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:10.653181+0000) 2019-10-18T01:20:11.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:11.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:11.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:11.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:12.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:12.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:12.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:12.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:13.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:13.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:13.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:13.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:14.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:20:14.101+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:14.107795+0000) 2019-10-18T01:20:14.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:14.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:14.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:14.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:15.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:15.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:15.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:15.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:15.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:15.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:16.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:16.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:16.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:16.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:17.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:17.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:17.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:17.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:18.210 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:20:18.205+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:18.206997+0000) 2019-10-18T01:20:18.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:18.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:18.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:18.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:18.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:18.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:18.716395+0000) 2019-10-18T01:20:19.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:19.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:19.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:19.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:20.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:20.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:20.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:20.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:20.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:20.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:20.656 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:20.649+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:20.653291+0000) 2019-10-18T01:20:21.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:21.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:21.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:21.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:22.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:22.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:22.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:22.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:23.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:23.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:23.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:23.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:24.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:20:24.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:24.107895+0000) 2019-10-18T01:20:24.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:24.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:24.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:24.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:25.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:25.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:25.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:25.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:25.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:25.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:26.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:26.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:26.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:26.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:27.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:27.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:27.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:27.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:28.210 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:20:28.205+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:28.207181+0000) 2019-10-18T01:20:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:28.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:28.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:28.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:28.713+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:28.716576+0000) 2019-10-18T01:20:29.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:29.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:29.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:29.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:30.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:30.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:30.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:30.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:30.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:30.405+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:30.656 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:30.649+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:30.653391+0000) 2019-10-18T01:20:31.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:31.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:31.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:31.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:32.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:32.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:32.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:32.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:33.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:33.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:33.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:33.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:34.110 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:20:34.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:34.107997+0000) 2019-10-18T01:20:34.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:34.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:34.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:34.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:35.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:35.404+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:35.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:35.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:35.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:35.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:36.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:36.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:36.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:37.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:37.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:37.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:37.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:38.210 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:20:38.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:38.207365+0000) 2019-10-18T01:20:38.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:38.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:38.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:38.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:38.719 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:38.712+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:38.716771+0000) 2019-10-18T01:20:38.720 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:38.712+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:38.717778+0000) 2019-10-18T01:20:39.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:39.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:39.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:39.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:40.410 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:40.404+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:40.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:40.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:40.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:40.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:40.656 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:40.652+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:40.653487+0000) 2019-10-18T01:20:40.657 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:40.652+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:40.654552+0000) 2019-10-18T01:20:41.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:41.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:41.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:41.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:42.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:42.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:42.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:42.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:43.220 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:43.212+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:43.217164+0000) 2019-10-18T01:20:43.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:43.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:43.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:43.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:44.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:20:44.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:44.108096+0000) 2019-10-18T01:20:44.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:44.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:44.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:44.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:45.156 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:45.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:45.153685+0000) 2019-10-18T01:20:45.412 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:45.404+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:45.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:45.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:45.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:45.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:46.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:46.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:46.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:46.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:47.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:47.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:47.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:47.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:48.210 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:20:48.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:48.207557+0000) 2019-10-18T01:20:48.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:48.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:48.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:48.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:49.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:49.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:49.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:49.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:50.412 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:50.408+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:50.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:50.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:50.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:50.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:51.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:51.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:51.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:51.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:52.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:52.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:52.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:52.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:53.220 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:20:53.212+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:53.217374+0000) 2019-10-18T01:20:53.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:53.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:53.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:53.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:54.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:20:54.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:54.108197+0000) 2019-10-18T01:20:54.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:54.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:54.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:54.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:55.156 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:20:55.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:55.153782+0000) 2019-10-18T01:20:55.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:20:55.408+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:20:55.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:55.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:55.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:55.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:56.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:56.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:56.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:56.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:57.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:57.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:57.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:57.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:58.217 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:20:58.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:20:58.207744+0000) 2019-10-18T01:20:58.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:58.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:58.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:58.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:20:59.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:59.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:20:59.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:20:59.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:00.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:00.408+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:00.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:00.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:00.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:00.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:01.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:01.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:01.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:01.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:02.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:02.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:02.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:02.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:03.220 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:21:03.212+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:03.217583+0000) 2019-10-18T01:21:03.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:03.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:03.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:03.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:04.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:21:04.104+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:04.108298+0000) 2019-10-18T01:21:04.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:04.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:04.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:04.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:05.156 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:21:05.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:05.153880+0000) 2019-10-18T01:21:05.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:05.408+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:05.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:05.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:05.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:05.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:06.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:06.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:06.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:06.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:07.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:07.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:07.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:07.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:08.211 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:21:08.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:08.207932+0000) 2019-10-18T01:21:08.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:08.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:08.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:08.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:09.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:09.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:09.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:09.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:10.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:10.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:10.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:10.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:10.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:10.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:11.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:11.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:11.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:11.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:12.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:12.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:12.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:12.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:13.220 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:21:13.212+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:13.217796+0000) 2019-10-18T01:21:13.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:13.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:13.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:13.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:14.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:21:14.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:14.108396+0000) 2019-10-18T01:21:14.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:14.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:14.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:14.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:15.156 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:21:15.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:15.153980+0000) 2019-10-18T01:21:15.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:15.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:15.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:15.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:15.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:15.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:16.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:16.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:16.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:16.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:17.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:17.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:17.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:17.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:18.211 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:21:18.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:18.208124+0000) 2019-10-18T01:21:18.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:18.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:18.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:18.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:19.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:19.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:19.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:19.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:20.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:20.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:20.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:20.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:20.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:20.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:21.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:21.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:21.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:21.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:22.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:22.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:22.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:22.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:23.221 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:21:23.212+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:23.218009+0000) 2019-10-18T01:21:23.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:23.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:23.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:23.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:24.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:21:24.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:24.108493+0000) 2019-10-18T01:21:24.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:24.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:24.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:24.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:25.156 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:21:25.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:25.154080+0000) 2019-10-18T01:21:25.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:25.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:25.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:25.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:25.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:25.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:26.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:26.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:26.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:26.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:27.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:27.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:27.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:27.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:28.211 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:21:28.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:28.208309+0000) 2019-10-18T01:21:28.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:28.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:28.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:28.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:29.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:29.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:29.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:29.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:30.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:30.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:30.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:30.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:30.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:30.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:31.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:31.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:31.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:31.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:32.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:32.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:32.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:32.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:33.221 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:21:33.216+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:33.218227+0000) 2019-10-18T01:21:33.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:33.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:33.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:33.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:34.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:21:34.103+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:34.108592+0000) 2019-10-18T01:21:34.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:34.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:34.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:34.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:35.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:21:35.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:35.154183+0000) 2019-10-18T01:21:35.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:35.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:35.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:35.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:35.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:35.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:36.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:36.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:36.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:36.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:37.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:37.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:37.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:37.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:38.211 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:21:38.204+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:38.208523+0000) 2019-10-18T01:21:38.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:38.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:38.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:38.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:39.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:39.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:39.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:39.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:40.421 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:40.415+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:40.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:40.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:40.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:40.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:41.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:41.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:41.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:41.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:42.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:42.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:42.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:42.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:43.221 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:21:43.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:43.218435+0000) 2019-10-18T01:21:43.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:43.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:43.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:43.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:44.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:21:44.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:44.108742+0000) 2019-10-18T01:21:44.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:44.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:44.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:44.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:45.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:21:45.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:45.154282+0000) 2019-10-18T01:21:45.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:45.414+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:45.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:45.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:45.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:45.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:46.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:46.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:46.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:46.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:47.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:47.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:47.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:47.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:48.211 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:21:48.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:48.208708+0000) 2019-10-18T01:21:48.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:48.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:48.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:48.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:49.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:49.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:50.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:50.414+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:50.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:50.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:50.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:50.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:51.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:51.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:51.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:51.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:52.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:52.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:52.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:52.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:53.221 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:21:53.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:53.218647+0000) 2019-10-18T01:21:53.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:53.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:53.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:53.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:54.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:21:54.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:54.108805+0000) 2019-10-18T01:21:54.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:54.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:54.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:54.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:55.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:21:55.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:55.154383+0000) 2019-10-18T01:21:55.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:21:55.414+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:21:55.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:55.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:55.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:55.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:56.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:56.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:56.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:56.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:57.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:57.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:57.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:57.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:58.212 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:21:58.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:21:58.208888+0000) 2019-10-18T01:21:58.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:58.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:58.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:58.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:21:59.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:59.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:21:59.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:21:59.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:00.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:00.418+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:00.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:00.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:00.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:00.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:01.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:01.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:01.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:01.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:02.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:02.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:02.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:02.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:03.221 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:22:03.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:03.218857+0000) 2019-10-18T01:22:03.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:03.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:03.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:03.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:04.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:22:04.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:04.108901+0000) 2019-10-18T01:22:04.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:04.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:04.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:04.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:05.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:22:05.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:05.154481+0000) 2019-10-18T01:22:05.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:05.418+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:05.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:05.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:05.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:05.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:06.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:06.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:06.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:06.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:07.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:07.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:07.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:07.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:08.212 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:22:08.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:08.209080+0000) 2019-10-18T01:22:08.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:08.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:08.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:08.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:09.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:09.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:09.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:09.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:10.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:10.418+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:10.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:10.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:10.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:10.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:11.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:11.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:11.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:11.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:12.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:12.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:12.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:12.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:13.222 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:22:13.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:13.219111+0000) 2019-10-18T01:22:13.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:13.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:13.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:13.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:14.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:22:14.102+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:14.108997+0000) 2019-10-18T01:22:14.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:14.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:14.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:14.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:15.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:22:15.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:15.154584+0000) 2019-10-18T01:22:15.427 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:15.418+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:15.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:15.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:15.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:15.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:16.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:16.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:16.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:16.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:17.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:17.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:17.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:17.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:18.212 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:22:18.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:18.209281+0000) 2019-10-18T01:22:18.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:18.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:18.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:18.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:19.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:19.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:19.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:19.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:20.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:20.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:20.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:20.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:20.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:20.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:21.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:21.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:21.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:21.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:22.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:22.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:22.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:22.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:23.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:22:23.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:23.219299+0000) 2019-10-18T01:22:23.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:23.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:23.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:23.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:24.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:22:24.101+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:24.109094+0000) 2019-10-18T01:22:24.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:24.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:24.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:24.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:25.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:22:25.149+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:25.154688+0000) 2019-10-18T01:22:25.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:25.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:25.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:25.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:25.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:25.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:26.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:26.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:26.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:26.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:27.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:27.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:27.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:27.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:28.212 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:22:28.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:28.209473+0000) 2019-10-18T01:22:28.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:28.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:28.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:28.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:29.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:29.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:29.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:29.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:30.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:30.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:30.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:30.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:30.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:30.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:31.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:31.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:31.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:31.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:32.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:32.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:32.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:32.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:33.222 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:22:33.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:33.219491+0000) 2019-10-18T01:22:33.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:33.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:33.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:33.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:34.111 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:22:34.101+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:34.109191+0000) 2019-10-18T01:22:34.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:34.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:34.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:34.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:35.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:22:35.149+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:35.154787+0000) 2019-10-18T01:22:35.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:35.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:35.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:35.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:35.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:35.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:36.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:36.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:36.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:36.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:37.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:37.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:37.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:37.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:38.212 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:22:38.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:38.209658+0000) 2019-10-18T01:22:38.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:38.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:38.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:38.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:39.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:39.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:39.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:39.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:40.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:40.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:40.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:40.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:40.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:40.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:41.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:41.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:41.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:41.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:42.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:42.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:42.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:42.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:43.222 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:22:43.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:43.219671+0000) 2019-10-18T01:22:43.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:43.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:43.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:43.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:44.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:22:44.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:44.109288+0000) 2019-10-18T01:22:44.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:44.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:44.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:44.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:45.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:22:45.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:45.154884+0000) 2019-10-18T01:22:45.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:45.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:45.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:45.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:45.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:45.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:46.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:46.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:46.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:46.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:47.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:47.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:47.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:47.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:48.213 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:22:48.203+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:48.209806+0000) 2019-10-18T01:22:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:48.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:48.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:49.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:49.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:49.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:49.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:50.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:50.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:50.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:50.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:50.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:50.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:51.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:51.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:51.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:51.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:52.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:52.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:53.222 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:22:53.215+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:53.219853+0000) 2019-10-18T01:22:53.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:53.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:53.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:53.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:54.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:22:54.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:54.109386+0000) 2019-10-18T01:22:54.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:54.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:54.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:54.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:55.157 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:22:55.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:55.154981+0000) 2019-10-18T01:22:55.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:22:55.428+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:22:55.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:55.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:55.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:55.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:56.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:56.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:56.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:56.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:57.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:57.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:57.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:57.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:58.213 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:22:58.202+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:22:58.210020+0000) 2019-10-18T01:22:58.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:58.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:58.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:58.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:22:59.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:59.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:22:59.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:22:59.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:00.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:00.428+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:00.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:00.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:00.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:00.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:01.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:01.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:01.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:01.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:02.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:02.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:02.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:02.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:03.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:23:03.214+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:03.220061+0000) 2019-10-18T01:23:03.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:03.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:03.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:03.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:04.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:23:04.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:04.109463+0000) 2019-10-18T01:23:04.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:04.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:04.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:04.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:05.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:23:05.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:05.155084+0000) 2019-10-18T01:23:05.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:05.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:05.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:05.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:05.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:05.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:06.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:06.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:06.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:06.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:07.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:07.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:07.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:07.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:08.213 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:23:08.206+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:08.210225+0000) 2019-10-18T01:23:08.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:08.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:08.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:08.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:09.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:09.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:09.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:09.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:10.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:10.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:10.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:10.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:10.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:10.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:11.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:11.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:11.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:11.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:12.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:12.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:12.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:12.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:13.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:23:13.214+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:13.220247+0000) 2019-10-18T01:23:14.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:13.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:14.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:13.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:14.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:23:14.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:14.109560+0000) 2019-10-18T01:23:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:15.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:15.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:15.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:23:15.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:15.155183+0000) 2019-10-18T01:23:15.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:15.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:16.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:16.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:16.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:16.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:16.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:16.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:16.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:16.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:17.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:17.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:17.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:17.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:18.213 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:23:18.206+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:18.210414+0000) 2019-10-18T01:23:18.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:18.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:18.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:18.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:19.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:19.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:19.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:19.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:20.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:20.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:20.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:20.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:20.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:20.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:22.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:22.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:22.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:22.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:23.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:22.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:23.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:22.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:23.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:23:23.214+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:23.220430+0000) 2019-10-18T01:23:23.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:23.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:23.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:23.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:24.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:23:24.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:24.109657+0000) 2019-10-18T01:23:24.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:24.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:24.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:24.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:25.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:23:25.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:25.155283+0000) 2019-10-18T01:23:25.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:25.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:25.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:25.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:26.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:26.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:26.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:26.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:27.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:27.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:27.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:27.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:28.213 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:23:28.206+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:28.210620+0000) 2019-10-18T01:23:28.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:28.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:28.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:28.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:30.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:30.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:30.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:30.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:30.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:30.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:30.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:30.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:30.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:30.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:31.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:31.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:31.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:31.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:33.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:32.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:33.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:32.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:33.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:23:33.214+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:33.220568+0000) 2019-10-18T01:23:33.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:33.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:33.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:33.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:34.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:23:34.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:34.109757+0000) 2019-10-18T01:23:34.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:34.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:34.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:34.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:35.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:23:35.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:35.155380+0000) 2019-10-18T01:23:35.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:35.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:35.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:35.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:35.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:35.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:36.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:36.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:36.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:36.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:37.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:37.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:37.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:37.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:38.213 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:23:38.206+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:38.210805+0000) 2019-10-18T01:23:38.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:38.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:38.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:38.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:39.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:39.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:39.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:39.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:40.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:40.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:40.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:40.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:40.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:40.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:41.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:41.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:41.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:41.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:42.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:42.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:42.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:42.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:43.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:23:43.214+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:43.220748+0000) 2019-10-18T01:23:44.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:43.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:44.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:43.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:44.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:23:44.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:44.109856+0000) 2019-10-18T01:23:44.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:44.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:44.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:44.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:45.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:23:45.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:45.155480+0000) 2019-10-18T01:23:45.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:45.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:45.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:45.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:45.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:45.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:46.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:46.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:46.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:46.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:47.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:47.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:47.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:47.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:48.214 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:23:48.206+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:48.211017+0000) 2019-10-18T01:23:48.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:48.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:48.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:48.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:49.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:49.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:49.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:49.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:50.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:50.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:50.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:50.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:51.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:51.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:51.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:51.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:52.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:52.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:52.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:52.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:53.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:23:53.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:53.220938+0000) 2019-10-18T01:23:53.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:53.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:53.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:53.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:54.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:23:54.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:54.109955+0000) 2019-10-18T01:23:54.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:54.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:54.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:54.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:55.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:23:55.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:55.155581+0000) 2019-10-18T01:23:55.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:23:55.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:23:55.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:55.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:55.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:55.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:56.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:56.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:56.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:56.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:57.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:57.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:57.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:57.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:58.214 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:23:58.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:23:58.211202+0000) 2019-10-18T01:23:58.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:58.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:58.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:58.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:23:59.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:59.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:23:59.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:23:59.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:00.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:00.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:00.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:00.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:00.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:00.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:01.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:01.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:01.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:01.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:02.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:02.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:02.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:02.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:03.224 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:24:03.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:03.221084+0000) 2019-10-18T01:24:03.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:03.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:03.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:03.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:04.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:24:04.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:04.110055+0000) 2019-10-18T01:24:04.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:04.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:04.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:04.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:05.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:24:05.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:05.155679+0000) 2019-10-18T01:24:05.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:05.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:05.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:05.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:05.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:05.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:06.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:06.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:06.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:06.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:07.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:07.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:07.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:07.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:08.214 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:24:08.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:08.211392+0000) 2019-10-18T01:24:08.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:08.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:08.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:08.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:09.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:09.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:09.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:09.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:10.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:10.438+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:10.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:10.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:10.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:10.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:11.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:11.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:11.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:11.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:12.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:12.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:12.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:12.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:13.224 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:24:13.217+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:13.221297+0000) 2019-10-18T01:24:13.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:13.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:13.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:13.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:14.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:24:14.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:14.110152+0000) 2019-10-18T01:24:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:14.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:14.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:15.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:24:15.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:15.155777+0000) 2019-10-18T01:24:15.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:15.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:15.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:15.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:15.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:15.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:16.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:16.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:16.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:16.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:17.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:17.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:17.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:17.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:18.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:24:18.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:18.211581+0000) 2019-10-18T01:24:18.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:18.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:18.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:18.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:19.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:19.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:20.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:20.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:20.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:20.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:20.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:20.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:21.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:21.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:21.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:21.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:22.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:22.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:22.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:22.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:23.224 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:24:23.217+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:23.221510+0000) 2019-10-18T01:24:23.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:23.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:23.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:23.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:24.112 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:24:24.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:24.110247+0000) 2019-10-18T01:24:24.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:24.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:24.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:24.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:25.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:24:25.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:25.155880+0000) 2019-10-18T01:24:25.450 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:25.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:25.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:25.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:26.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:26.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:26.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:26.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:27.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:27.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:27.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:27.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:28.214 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:24:28.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:28.211766+0000) 2019-10-18T01:24:28.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:28.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:28.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:28.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:29.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:29.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:29.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:29.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:30.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:30.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:30.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:30.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:30.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:30.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:31.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:31.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:31.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:31.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:32.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:33.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:32.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:33.224 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:24:33.217+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:33.221648+0000) 2019-10-18T01:24:33.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:33.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:33.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:33.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:34.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:24:34.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:34.110344+0000) 2019-10-18T01:24:34.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:34.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:34.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:34.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:35.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:24:35.150+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:35.155977+0000) 2019-10-18T01:24:35.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:35.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:35.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:35.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:35.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:35.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:36.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:36.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:36.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:36.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:37.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:37.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:37.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:37.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:38.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:24:38.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:38.211947+0000) 2019-10-18T01:24:38.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:38.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:38.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:38.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:39.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:39.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:39.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:39.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:40.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:40.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:40.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:40.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:40.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:40.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:41.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:41.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:41.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:41.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:42.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:42.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:42.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:42.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:43.224 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:24:43.217+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:43.221806+0000) 2019-10-18T01:24:43.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:43.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:43.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:43.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:44.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:24:44.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:44.110444+0000) 2019-10-18T01:24:44.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:44.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:44.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:44.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:45.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:24:45.149+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:45.156075+0000) 2019-10-18T01:24:45.454 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:45.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:45.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:45.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:45.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:45.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:46.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:46.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:46.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:46.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:47.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:47.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:47.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:47.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:48.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:24:48.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:48.212139+0000) 2019-10-18T01:24:48.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:48.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:48.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:48.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:49.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:49.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:49.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:49.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:50.454 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:50.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:50.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:50.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:50.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:50.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:51.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:51.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:51.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:51.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:52.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:52.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:52.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:52.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:53.225 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:24:53.217+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:53.221984+0000) 2019-10-18T01:24:53.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:53.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:53.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:53.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:54.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:24:54.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:54.110539+0000) 2019-10-18T01:24:54.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:54.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:54.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:54.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:55.158 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:24:55.149+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:55.156178+0000) 2019-10-18T01:24:55.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:24:55.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:24:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:55.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:55.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:56.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:56.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:56.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:56.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:57.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:57.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:57.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:57.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:58.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:24:58.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:24:58.212313+0000) 2019-10-18T01:24:58.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:58.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:58.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:58.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:24:59.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:59.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:24:59.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:24:59.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:00.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:00.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:00.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:00.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:00.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:00.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:01.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:01.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:01.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:01.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:02.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:02.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:02.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:02.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:03.225 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:25:03.221+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:03.222169+0000) 2019-10-18T01:25:03.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:03.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:03.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:03.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:04.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:25:04.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:04.110636+0000) 2019-10-18T01:25:04.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:04.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:04.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:04.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:05.159 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:05.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:05.156276+0000) 2019-10-18T01:25:05.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:05.453+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:05.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:05.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:05.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:05.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:06.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:06.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:06.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:06.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:07.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:07.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:07.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:07.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:08.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:25:08.209+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:08.212506+0000) 2019-10-18T01:25:08.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:08.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:08.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:08.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:09.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:09.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:09.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:09.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:10.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:10.453+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:10.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:10.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:10.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:10.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:11.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:11.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:11.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:11.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:12.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:12.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:14:46.285206+0000 (oldest deadline 2019-10-18T01:15:06.285206+0000) 2019-10-18T01:25:12.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:12.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:13.225 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:25:13.221+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:13.222339+0000) 2019-10-18T01:25:13.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:13.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:14.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:25:14.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:14.110734+0000) 2019-10-18T01:25:14.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:14.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:15.159 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:15.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:15.156378+0000) 2019-10-18T01:25:15.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:15.453+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:15.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:15.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:16.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:16.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:17.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:17.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:18.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:25:18.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:18.212688+0000) 2019-10-18T01:25:18.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:18.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:19.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:19.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:20.460 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:20.452+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:20.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:20.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:21.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:21.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:22.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:22.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:23.225 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:25:23.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:23.222515+0000) 2019-10-18T01:25:23.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:23.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:24.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:25:24.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:24.110835+0000) 2019-10-18T01:25:24.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:24.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:25.159 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:25.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:25.156477+0000) 2019-10-18T01:25:25.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:25.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:25.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:25.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:26.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:26.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:27.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:27.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:28.215 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:25:28.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:28.212872+0000) 2019-10-18T01:25:28.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:28.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:29.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:29.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:30.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:30.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:30.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:30.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:31.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:31.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:32.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:32.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:33.225 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:25:33.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:33.222687+0000) 2019-10-18T01:25:33.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:33.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:34.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:25:34.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:34.110929+0000) 2019-10-18T01:25:34.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:34.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:35.159 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:35.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:35.156574+0000) 2019-10-18T01:25:35.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:35.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:35.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:35.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:35.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:35.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:36.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:36.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:36.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:36.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:37.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:37.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:37.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:37.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:38.216 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:25:38.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:38.213072+0000) 2019-10-18T01:25:38.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:38.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:38.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:38.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:39.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:39.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:39.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:39.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:40.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:40.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:40.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:40.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:40.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:40.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:41.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:41.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:41.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:41.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:42.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:42.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:42.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:42.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:43.225 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:25:43.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:43.222880+0000) 2019-10-18T01:25:43.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:43.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:43.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:43.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:44.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:25:44.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:44.111039+0000) 2019-10-18T01:25:44.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:44.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:44.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:44.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:45.159 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:45.152+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:45.156667+0000) 2019-10-18T01:25:45.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:45.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:45.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:45.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:45.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:45.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:46.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:46.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:46.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:46.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:47.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:47.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:47.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:47.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:48.216 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:25:48.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:48.213254+0000) 2019-10-18T01:25:48.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:48.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:48.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:48.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:49.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:49.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:49.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:49.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:50.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:50.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:50.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:50.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:50.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:50.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:51.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:51.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:51.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:51.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:52.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:52.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:52.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:52.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:53.229 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:25:53.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:53.223090+0000) 2019-10-18T01:25:53.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:53.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:53.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:53.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:54.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:25:54.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:54.111149+0000) 2019-10-18T01:25:54.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:54.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:54.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:54.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:55.159 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:55.151+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:55.156760+0000) 2019-10-18T01:25:55.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:55.155+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:55.157635+0000) 2019-10-18T01:25:55.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:25:55.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:25:55.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:55.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:55.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:55.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:56.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:56.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:56.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:56.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:57.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:57.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:57.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:57.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:58.216 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:25:58.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:58.213457+0000) 2019-10-18T01:25:58.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:58.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:58.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:58.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:59.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:59.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:25:59.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:25:59.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:25:59.659 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:25:59.651+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:25:59.656971+0000) 2019-10-18T01:26:00.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:00.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:00.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:00.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:01.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:01.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:01.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:01.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:02.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:02.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:02.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:02.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:03.226 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:26:03.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:03.223306+0000) 2019-10-18T01:26:03.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:03.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:03.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:03.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:04.113 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:26:04.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:04.111241+0000) 2019-10-18T01:26:04.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:04.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:04.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:04.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:05.467 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:05.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:05.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:05.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:05.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:05.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:06.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:06.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:06.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:06.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:07.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:07.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:07.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:07.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:08.216 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:26:08.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:08.213642+0000) 2019-10-18T01:26:08.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:08.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:08.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:08.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:09.659 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:09.651+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:09.657073+0000) 2019-10-18T01:26:09.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:09.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:09.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:09.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:10.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:10.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:10.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:10.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:10.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:10.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:11.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:11.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:11.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:11.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:12.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:12.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:12.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:12.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:13.226 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:26:13.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:13.223492+0000) 2019-10-18T01:26:13.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:13.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:13.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:13.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:14.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:26:14.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:14.111337+0000) 2019-10-18T01:26:14.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:14.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:14.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:14.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:15.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:15.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:15.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:15.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:15.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:15.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:16.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:16.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:16.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:16.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:17.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:17.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:17.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:17.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:18.217 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:26:18.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:18.213825+0000) 2019-10-18T01:26:18.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:18.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:18.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:18.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:19.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:19.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:19.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:19.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:19.659 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:19.651+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:19.657172+0000) 2019-10-18T01:26:20.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:20.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:20.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:20.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:20.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:20.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:21.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:21.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:21.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:21.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:22.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:22.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:22.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:22.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:23.226 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:26:23.220+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:23.223669+0000) 2019-10-18T01:26:23.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:23.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:23.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:23.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:24.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:26:24.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:24.111437+0000) 2019-10-18T01:26:24.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:24.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:24.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:24.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:25.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:25.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:25.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:25.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:25.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:25.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:26.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:26.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:26.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:26.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:27.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:27.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:27.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:27.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:28.217 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:26:28.208+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:28.214013+0000) 2019-10-18T01:26:28.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:28.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:28.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:28.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:29.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:29.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:29.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:29.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:29.659 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:29.651+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:29.657266+0000) 2019-10-18T01:26:30.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:30.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:30.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:30.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:30.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:30.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:31.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:31.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:31.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:31.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:32.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:32.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:32.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:32.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:33.226 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:26:33.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:33.223852+0000) 2019-10-18T01:26:33.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:33.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:33.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:33.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:34.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:26:34.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:34.111533+0000) 2019-10-18T01:26:34.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:34.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:34.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:34.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:35.472 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:35.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:35.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:35.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:35.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:35.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:36.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:36.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:36.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:36.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:37.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:37.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:37.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:37.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:38.217 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:26:38.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:38.214221+0000) 2019-10-18T01:26:38.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:38.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:38.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:38.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:39.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:39.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:39.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:39.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:39.660 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:39.650+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:39.657359+0000) 2019-10-18T01:26:39.661 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:39.654+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:39.658988+0000) 2019-10-18T01:26:40.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:40.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:40.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:40.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:40.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:40.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:41.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:41.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:41.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:41.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:42.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:42.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:42.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:42.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:43.227 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:26:43.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:43.224039+0000) 2019-10-18T01:26:43.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:43.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:43.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:43.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:44.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:26:44.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:44.111633+0000) 2019-10-18T01:26:44.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:44.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:44.157572+0000) 2019-10-18T01:26:44.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:44.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:44.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:44.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:45.474 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:45.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:45.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:45.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:45.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:45.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:46.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:46.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:46.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:46.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:47.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:47.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:47.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:47.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:48.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:26:48.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:48.214402+0000) 2019-10-18T01:26:48.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:48.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:48.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:48.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:49.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:49.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:49.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:49.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:50.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:50.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:50.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:50.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:50.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:50.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:51.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:51.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:51.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:51.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:52.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:52.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:52.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:52.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:53.227 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:26:53.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:53.224224+0000) 2019-10-18T01:26:53.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:53.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:53.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:53.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:54.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:26:54.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:54.111736+0000) 2019-10-18T01:26:54.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:26:54.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:54.157667+0000) 2019-10-18T01:26:54.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:54.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:54.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:54.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:55.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:26:55.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:26:55.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:55.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:55.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:55.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:56.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:56.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:56.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:56.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:57.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:57.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:57.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:57.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:58.217 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:26:58.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:26:58.214593+0000) 2019-10-18T01:26:58.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:58.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:58.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:58.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:26:59.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:59.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:26:59.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:26:59.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:00.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:00.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:00.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:00.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:00.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:00.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:01.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:01.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:01.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:01.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:02.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:02.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:02.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:02.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:03.227 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:27:03.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:03.224437+0000) 2019-10-18T01:27:03.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:03.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:03.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:03.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:04.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:27:04.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:04.111831+0000) 2019-10-18T01:27:04.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:27:04.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:04.157761+0000) 2019-10-18T01:27:04.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:04.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:04.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:04.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:05.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:05.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:05.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:05.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:05.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:05.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:06.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:06.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:06.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:06.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:07.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:07.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:07.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:07.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:08.217 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:27:08.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:08.214821+0000) 2019-10-18T01:27:08.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:08.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:08.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:08.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:09.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:09.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:09.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:09.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:10.479 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:10.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:10.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:10.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:10.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:10.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:11.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:11.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:11.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:11.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:12.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:12.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:12.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:12.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:13.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:27:13.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:13.224576+0000) 2019-10-18T01:27:13.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:13.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:13.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:13.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:14.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:27:14.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:14.111929+0000) 2019-10-18T01:27:14.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:27:14.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:14.157852+0000) 2019-10-18T01:27:14.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:14.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:14.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:14.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:15.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:15.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:15.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:15.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:15.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:15.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:16.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:16.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:16.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:16.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:17.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:17.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:17.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:17.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:18.218 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:27:18.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:18.215009+0000) 2019-10-18T01:27:18.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:18.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:18.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:18.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:19.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:19.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:19.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:19.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:20.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:20.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:20.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:20.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:20.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:20.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:21.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:21.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:22.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:22.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:22.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:22.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:23.227 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:27:23.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:23.224717+0000) 2019-10-18T01:27:23.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:23.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:23.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:23.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:24.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:27:24.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:24.112028+0000) 2019-10-18T01:27:24.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:27:24.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:24.157950+0000) 2019-10-18T01:27:24.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:24.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:24.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:24.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:25.481 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:25.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:25.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:25.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:25.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:25.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:26.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:26.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:26.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:26.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:27.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:27.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:28.218 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:27:28.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:28.215153+0000) 2019-10-18T01:27:28.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:28.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:28.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:28.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:29.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:29.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:29.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:29.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:30.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:30.477+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:30.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:30.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:30.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:30.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:31.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:31.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:31.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:31.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:32.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:32.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:32.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:32.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:33.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:27:33.219+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:33.224888+0000) 2019-10-18T01:27:33.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:33.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:33.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:33.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:34.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:27:34.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:34.112124+0000) 2019-10-18T01:27:34.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:27:34.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:34.158044+0000) 2019-10-18T01:27:34.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:34.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:34.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:34.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:35.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:35.477+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:35.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:35.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:35.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:35.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:36.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:36.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:36.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:36.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:37.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:37.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:37.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:37.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:38.218 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:27:38.211+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:38.215348+0000) 2019-10-18T01:27:38.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:38.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:38.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:38.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:39.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:39.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:39.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:39.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:40.484 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:40.481+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:40.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:40.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:40.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:40.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:41.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:41.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:41.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:41.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:42.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:42.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:42.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:42.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:43.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:27:43.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:43.225034+0000) 2019-10-18T01:27:43.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:43.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:43.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:43.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:44.114 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:27:44.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:44.112221+0000) 2019-10-18T01:27:44.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:27:44.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:44.158139+0000) 2019-10-18T01:27:44.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:44.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:44.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:44.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:45.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:45.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:45.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:45.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:45.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:45.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:46.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:46.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:46.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:46.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:47.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:47.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:47.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:47.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:48.218 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:27:48.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:48.215536+0000) 2019-10-18T01:27:48.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:48.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:48.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:48.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:49.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:49.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:49.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:49.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:50.486 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:50.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:50.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:50.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:50.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:50.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:51.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:51.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:51.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:51.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:52.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:52.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:52.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:52.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:53.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:27:53.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:53.225246+0000) 2019-10-18T01:27:53.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:53.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:53.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:53.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:54.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:27:54.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:54.112318+0000) 2019-10-18T01:27:54.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:27:54.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:54.158233+0000) 2019-10-18T01:27:54.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:54.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:54.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:54.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:55.486 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:27:55.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:27:55.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:55.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:55.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:55.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:56.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:56.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:56.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:56.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:57.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:57.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:57.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:57.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:58.218 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:27:58.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:27:58.215723+0000) 2019-10-18T01:27:58.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:58.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:58.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:58.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:27:59.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:59.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:27:59.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:27:59.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:00.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:00.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:00.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:00.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:00.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:00.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:01.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:01.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:01.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:01.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:02.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:02.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:02.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:02.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:03.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:28:03.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:03.225405+0000) 2019-10-18T01:28:03.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:03.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:03.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:03.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:04.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:28:04.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:04.112413+0000) 2019-10-18T01:28:04.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:28:04.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:04.158329+0000) 2019-10-18T01:28:04.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:04.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:04.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:04.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:05.489 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:05.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:05.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:05.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:05.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:05.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:06.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:06.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:06.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:06.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:07.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:07.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:07.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:07.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:08.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:28:08.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:08.215918+0000) 2019-10-18T01:28:08.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:08.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:08.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:08.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:09.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:09.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:09.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:09.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:10.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:10.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:10.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:10.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:10.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:10.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:11.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:11.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:11.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:11.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:12.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:12.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:12.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:12.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:13.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:28:13.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:13.225554+0000) 2019-10-18T01:28:13.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:13.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:13.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:13.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:14.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:28:14.108+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:14.112512+0000) 2019-10-18T01:28:14.160 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:28:14.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:14.158423+0000) 2019-10-18T01:28:14.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:14.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:14.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:14.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:15.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:15.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:15.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:15.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:15.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:15.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:16.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:16.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:17.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:17.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:17.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:17.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:18.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:28:18.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:18.216129+0000) 2019-10-18T01:28:18.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:18.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:18.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:18.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:19.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:19.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:19.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:19.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:20.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:20.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:20.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:20.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:20.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:20.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:21.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:21.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:21.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:21.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:22.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:22.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:22.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:22.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:23.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:28:23.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:23.225735+0000) 2019-10-18T01:28:23.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:23.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:23.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:23.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:24.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:28:24.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:24.112612+0000) 2019-10-18T01:28:24.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:28:24.155+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:24.158524+0000) 2019-10-18T01:28:24.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:24.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:24.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:24.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:25.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:25.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:25.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:25.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:25.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:25.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:26.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:26.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:26.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:26.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:27.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:27.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:27.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:27.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:28.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:28:28.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:28.216319+0000) 2019-10-18T01:28:28.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:28.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:28.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:28.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:29.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:29.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:29.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:29.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:30.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:30.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:30.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:30.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:30.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:30.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:31.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:31.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:31.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:31.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:32.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:32.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:32.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:32.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:33.228 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:28:33.218+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:33.225947+0000) 2019-10-18T01:28:33.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:33.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:33.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:33.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:34.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:28:34.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:34.112708+0000) 2019-10-18T01:28:34.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:28:34.155+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:34.158620+0000) 2019-10-18T01:28:34.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:34.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:34.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:34.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:35.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:35.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:35.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:35.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:35.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:35.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:36.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:36.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:36.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:36.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:37.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:37.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:37.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:37.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:38.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:28:38.210+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:38.216508+0000) 2019-10-18T01:28:38.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:38.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:38.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:38.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:39.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:39.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:39.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:39.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:40.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:40.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:40.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:40.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:40.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:40.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:41.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:41.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:41.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:41.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:42.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:42.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:42.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:42.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:43.229 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:28:43.222+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:43.226161+0000) 2019-10-18T01:28:43.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:43.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:43.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:43.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:44.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:28:44.107+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:44.112809+0000) 2019-10-18T01:28:44.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:28:44.155+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:44.158716+0000) 2019-10-18T01:28:44.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:44.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:44.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:44.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:45.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:45.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:45.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:45.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:45.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:45.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:46.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:46.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:46.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:46.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:47.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:47.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:47.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:47.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:48.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:28:48.214+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:48.216729+0000) 2019-10-18T01:28:48.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:48.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:48.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:48.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:49.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:49.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:49.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:49.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:50.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:50.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:50.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:50.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:50.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:50.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:51.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:51.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:51.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:51.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:52.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:52.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:52.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:52.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:53.229 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:28:53.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:53.226349+0000) 2019-10-18T01:28:53.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:53.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:53.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:53.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:54.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:28:54.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:54.112912+0000) 2019-10-18T01:28:54.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:28:54.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:54.158817+0000) 2019-10-18T01:28:54.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:54.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:54.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:54.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:55.497 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:28:55.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:28:55.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:55.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:55.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:55.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:56.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:56.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:56.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:56.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:57.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:57.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:57.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:57.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:58.219 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:28:58.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:28:58.216915+0000) 2019-10-18T01:28:58.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:58.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:58.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:58.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:28:59.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:59.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:28:59.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:28:59.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:00.498 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:00.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:00.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:00.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:00.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:00.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:01.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:01.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:01.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:01.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:02.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:02.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:02.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:02.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:03.229 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:29:03.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:03.226499+0000) 2019-10-18T01:29:03.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:03.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:03.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:03.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:04.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:29:04.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:04.113007+0000) 2019-10-18T01:29:04.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:29:04.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:04.158915+0000) 2019-10-18T01:29:04.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:04.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:04.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:04.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:05.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:05.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:05.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:05.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:05.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:05.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:06.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:06.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:06.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:06.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:07.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:07.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:07.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:07.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:08.220 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:29:08.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:08.217137+0000) 2019-10-18T01:29:08.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:08.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:08.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:08.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:09.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:09.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:09.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:09.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:10.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:10.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:10.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:10.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:10.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:10.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:11.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:11.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:11.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:11.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:12.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:12.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:12.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:12.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:13.229 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:29:13.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:13.226677+0000) 2019-10-18T01:29:13.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:13.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:13.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:13.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:14.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:29:14.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:14.113107+0000) 2019-10-18T01:29:14.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:29:14.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:14.159012+0000) 2019-10-18T01:29:14.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:14.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:14.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:14.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:15.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:15.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:15.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:15.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:15.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:15.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:16.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:16.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:16.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:16.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:17.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:17.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:17.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:17.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:18.220 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:29:18.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:18.217317+0000) 2019-10-18T01:29:18.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:18.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:18.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:18.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:19.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:19.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:19.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:19.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:20.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:20.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:20.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:20.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:20.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:20.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:21.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:21.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:21.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:21.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:22.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:22.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:22.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:23.229 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:29:23.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:23.226890+0000) 2019-10-18T01:29:23.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:23.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:23.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:23.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:24.115 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:29:24.106+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:24.113207+0000) 2019-10-18T01:29:24.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:29:24.154+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:24.159111+0000) 2019-10-18T01:29:24.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:24.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:24.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:24.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:25.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:25.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:25.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:25.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:25.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:25.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:26.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:26.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:26.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:26.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:27.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:27.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:27.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:27.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:28.220 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:29:28.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:28.217509+0000) 2019-10-18T01:29:28.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:28.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:28.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:28.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:29.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:29.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:29.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:29.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:30.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:30.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:30.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:30.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:30.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:30.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:31.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:31.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:31.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:31.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:32.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:32.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:32.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:32.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:33.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:29:33.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:33.227068+0000) 2019-10-18T01:29:33.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:33.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:33.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:33.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:34.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:29:34.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:34.113300+0000) 2019-10-18T01:29:34.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:29:34.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:34.159206+0000) 2019-10-18T01:29:34.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:34.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:34.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:34.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:35.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:35.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:35.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:35.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:35.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:35.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:36.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:36.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:36.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:36.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:37.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:37.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:37.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:37.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:38.220 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:29:38.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:38.217684+0000) 2019-10-18T01:29:38.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:38.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:38.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:38.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:39.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:39.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:39.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:39.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:40.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:40.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:40.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:40.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:40.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:40.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:41.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:41.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:41.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:41.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:42.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:42.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:42.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:42.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:43.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:29:43.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:43.227281+0000) 2019-10-18T01:29:43.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:43.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:43.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:43.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:44.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:29:44.105+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:44.113397+0000) 2019-10-18T01:29:44.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:29:44.153+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:44.159299+0000) 2019-10-18T01:29:44.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:44.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:44.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:44.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:45.506 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:45.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:45.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:45.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:45.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:45.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:46.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:46.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:47.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:47.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:47.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:47.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:48.220 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:29:48.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:48.217826+0000) 2019-10-18T01:29:48.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:48.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:48.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:48.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:49.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:49.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:49.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:49.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:50.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:50.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:50.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:50.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:50.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:50.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:51.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:51.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:51.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:51.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:52.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:52.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:52.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:52.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:53.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:29:53.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:53.227493+0000) 2019-10-18T01:29:53.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:53.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:53.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:53.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:54.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:29:54.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:54.113467+0000) 2019-10-18T01:29:54.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:29:54.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:54.159393+0000) 2019-10-18T01:29:54.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:54.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:54.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:54.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:55.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:29:55.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:29:55.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:55.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:55.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:55.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:56.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:56.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:56.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:56.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:57.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:57.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:57.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:57.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:58.221 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:29:58.213+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:29:58.217965+0000) 2019-10-18T01:29:58.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:58.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:58.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:58.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:29:59.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:59.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:29:59.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:29:59.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:00.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:00.505+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:00.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:00.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:00.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:00.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:01.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:01.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:01.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:01.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:02.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:02.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:02.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:02.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:03.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:30:03.225+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:03.227670+0000) 2019-10-18T01:30:03.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:03.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:03.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:03.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:04.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:30:04.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:04.113559+0000) 2019-10-18T01:30:04.161 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:04.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:04.159490+0000) 2019-10-18T01:30:04.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:04.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:04.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:04.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:05.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:05.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:05.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:05.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:05.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:05.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:06.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:06.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:06.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:06.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:07.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:07.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:07.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:07.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:08.221 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:08.212+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:08.218105+0000) 2019-10-18T01:30:08.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:08.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:08.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:08.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:09.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:09.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:09.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:09.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:10.510 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:10.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:10.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:10.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:10.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:10.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:11.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:11.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:11.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:11.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:12.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:12.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:12.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:12.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:13.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:30:13.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:13.227837+0000) 2019-10-18T01:30:13.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:13.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:13.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:13.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:14.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:30:14.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:14.113657+0000) 2019-10-18T01:30:14.162 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:14.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:14.159584+0000) 2019-10-18T01:30:14.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:14.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:14.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:14.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:15.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:15.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:15.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:15.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:15.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:15.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:16.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:16.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:16.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:16.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:17.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:17.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:17.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:17.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:18.221 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:18.216+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:18.218312+0000) 2019-10-18T01:30:18.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:18.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:18.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:18.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:19.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:19.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:19.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:19.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:20.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:20.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:20.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:20.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:20.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:20.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:21.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:21.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:21.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:21.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:22.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:22.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:22.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:22.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:23.230 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:30:23.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:23.227982+0000) 2019-10-18T01:30:23.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:23.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:23.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:23.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:24.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:30:24.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:24.113766+0000) 2019-10-18T01:30:24.162 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:24.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:24.159677+0000) 2019-10-18T01:30:24.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:24.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:24.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:24.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:25.513 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:25.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:25.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:25.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:25.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:25.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:26.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:26.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:26.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:26.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:27.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:27.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:27.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:27.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:28.221 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:28.216+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:28.218468+0000) 2019-10-18T01:30:28.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:28.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:28.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:28.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:29.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:29.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:29.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:29.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:30.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:30.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:30.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:30.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:30.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:30.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:31.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:31.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:31.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:31.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:32.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:32.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:32.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:32.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:33.231 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:30:33.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:33.228124+0000) 2019-10-18T01:30:33.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:33.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:33.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:33.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:34.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:30:34.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:34.113859+0000) 2019-10-18T01:30:34.162 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:34.156+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:34.159768+0000) 2019-10-18T01:30:34.163 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:34.156+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:34.160756+0000) 2019-10-18T01:30:34.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:34.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:34.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:34.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:35.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:35.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:35.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:35.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:35.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:35.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:36.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:36.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:36.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:36.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:37.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:37.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:37.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:37.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:38.221 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:38.216+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:38.218645+0000) 2019-10-18T01:30:38.222 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:38.216+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:38.219494+0000) 2019-10-18T01:30:38.662 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:38.656+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:38.659962+0000) 2019-10-18T01:30:38.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:38.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:38.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:38.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:39.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:39.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:39.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:39.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:40.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:40.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:40.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:40.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:40.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:40.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:41.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:41.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:41.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:41.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:42.722 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:42.716+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:42.719034+0000) 2019-10-18T01:30:42.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:42.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:42.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:42.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:43.232 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:30:43.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:43.228337+0000) 2019-10-18T01:30:43.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:43.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:43.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:43.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:44.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:30:44.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:44.113957+0000) 2019-10-18T01:30:44.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:44.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:44.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:44.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:45.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:45.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:45.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:45.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:45.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:45.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:46.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:46.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:46.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:46.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:47.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:47.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:47.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:47.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:48.662 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:48.655+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:48.660060+0000) 2019-10-18T01:30:48.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:48.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:48.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:48.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:49.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:49.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:49.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:49.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:50.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:50.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:50.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:50.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:50.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:50.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:51.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:51.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:51.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:51.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:52.722 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:30:52.716+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:52.719212+0000) 2019-10-18T01:30:52.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:52.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:52.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:52.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:53.231 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:30:53.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:53.228520+0000) 2019-10-18T01:30:53.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:53.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:53.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:53.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:54.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:30:54.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:54.114054+0000) 2019-10-18T01:30:54.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:54.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:54.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:54.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:55.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:30:55.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:30:55.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:55.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:55.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:55.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:56.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:56.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:56.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:56.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:57.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:57.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:57.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:57.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:58.662 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:30:58.655+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:30:58.660156+0000) 2019-10-18T01:30:58.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:58.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:58.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:58.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:30:59.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:59.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:30:59.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:30:59.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:00.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:00.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:00.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:00.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:00.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:00.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:01.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:01.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:01.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:01.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:02.722 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:31:02.716+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:02.719406+0000) 2019-10-18T01:31:02.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:02.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:02.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:02.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:03.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:31:03.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:03.228707+0000) 2019-10-18T01:31:03.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:03.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:03.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:03.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:04.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:31:04.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:04.114148+0000) 2019-10-18T01:31:04.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:04.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:04.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:04.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:05.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:05.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:05.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:05.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:05.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:05.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:06.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:06.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:06.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:06.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:07.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:07.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:07.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:07.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:08.662 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:31:08.655+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:08.660258+0000) 2019-10-18T01:31:08.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:08.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:08.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:08.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:09.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:09.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:09.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:09.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:10.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:10.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:10.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:10.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:10.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:10.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:11.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:11.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:11.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:11.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:12.722 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:31:12.716+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:12.719589+0000) 2019-10-18T01:31:12.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:12.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:12.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:12.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:13.231 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:31:13.224+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:13.228918+0000) 2019-10-18T01:31:13.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:13.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:13.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:13.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:14.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:31:14.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:14.114247+0000) 2019-10-18T01:31:14.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:14.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:14.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:14.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:15.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:15.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:15.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:15.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:15.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:15.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:16.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:16.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:16.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:16.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:17.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:17.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:17.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:17.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:18.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:31:18.654+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:18.660357+0000) 2019-10-18T01:31:18.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:18.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:18.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:18.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:19.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:19.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:19.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:19.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:20.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:20.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:20.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:20.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:20.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:20.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:21.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:21.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:21.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:21.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:22.722 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:31:22.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:22.719802+0000) 2019-10-18T01:31:22.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:22.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:22.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:22.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:23.232 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:31:23.223+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:23.229098+0000) 2019-10-18T01:31:23.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:23.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:23.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:23.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:24.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:31:24.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:24.114345+0000) 2019-10-18T01:31:24.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:24.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:24.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:24.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:25.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:25.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:25.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:25.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:25.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:25.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:26.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:26.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:26.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:26.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:27.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:27.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:27.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:27.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:28.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:31:28.654+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:28.660455+0000) 2019-10-18T01:31:28.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:28.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:28.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:28.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:29.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:29.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:29.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:29.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:30.524 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:30.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:30.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:30.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:30.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:30.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:31.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:31.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:31.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:31.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:32.722 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:31:32.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:32.720008+0000) 2019-10-18T01:31:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:32.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:32.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:32.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:33.232 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:31:33.223+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:33.229333+0000) 2019-10-18T01:31:33.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:33.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:33.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:33.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:34.116 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:31:34.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:34.114440+0000) 2019-10-18T01:31:34.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:34.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:34.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:34.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:35.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:35.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:35.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:35.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:35.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:35.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:36.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:36.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:36.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:36.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:37.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:37.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:37.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:37.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:38.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:31:38.654+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:38.660557+0000) 2019-10-18T01:31:38.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:38.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:38.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:38.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:39.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:39.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:39.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:39.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:40.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:40.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:40.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:40.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:40.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:40.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:41.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:41.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:41.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:41.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:42.723 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:31:42.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:42.720200+0000) 2019-10-18T01:31:42.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:42.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:42.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:42.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:43.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:31:43.223+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:43.229513+0000) 2019-10-18T01:31:43.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:43.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:43.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:43.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:44.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:31:44.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:44.114539+0000) 2019-10-18T01:31:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:44.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:44.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:45.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:45.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:45.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:45.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:45.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:45.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:46.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:46.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:46.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:46.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:47.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:47.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:47.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:47.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:48.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:31:48.654+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:48.660654+0000) 2019-10-18T01:31:48.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:48.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:48.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:48.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:50.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:50.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:50.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:50.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:50.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:50.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:51.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:51.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:51.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:51.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:52.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:52.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:52.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:52.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:52.723 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:31:52.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:52.720390+0000) 2019-10-18T01:31:53.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:53.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:53.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:53.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:53.232 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:31:53.223+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:53.229692+0000) 2019-10-18T01:31:54.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:54.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:54.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:54.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:54.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:31:54.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:54.114650+0000) 2019-10-18T01:31:55.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:55.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:55.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:55.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:55.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:31:55.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:31:56.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:56.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:56.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:56.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:57.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:57.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:57.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:57.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:58.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:58.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:58.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:58.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:31:58.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:31:58.653+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:31:58.660757+0000) 2019-10-18T01:31:59.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:59.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:31:59.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:31:59.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:00.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:00.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:00.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:00.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:00.529 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:00.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:01.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:01.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:01.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:01.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:02.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:02.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:02.723 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:32:02.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:02.720571+0000) 2019-10-18T01:32:02.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:02.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:02.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:02.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:03.235 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:32:03.223+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:03.229883+0000) 2019-10-18T01:32:03.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:03.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:03.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:03.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:04.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:32:04.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:04.114745+0000) 2019-10-18T01:32:04.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:04.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:04.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:04.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:05.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:05.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:05.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:05.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:05.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:05.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:06.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:06.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:06.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:06.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:07.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:07.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:07.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:07.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:08.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:32:08.653+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:08.660856+0000) 2019-10-18T01:32:08.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:08.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:08.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:08.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:09.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:09.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:09.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:09.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:10.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:10.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:11.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:11.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:11.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:11.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:12.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:12.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:12.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:12.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:12.723 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:32:12.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:12.720788+0000) 2019-10-18T01:32:13.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:13.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:13.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:13.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:13.232 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:32:13.223+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:13.230061+0000) 2019-10-18T01:32:14.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:14.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:14.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:14.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:14.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:32:14.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:14.114841+0000) 2019-10-18T01:32:15.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:15.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:15.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:15.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:15.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:15.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:16.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:16.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:16.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:16.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:17.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:17.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:17.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:17.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:18.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:18.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:18.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:18.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:18.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:32:18.657+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:18.660954+0000) 2019-10-18T01:32:19.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:19.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:19.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:19.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:20.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:20.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:20.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:20.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:21.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:21.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:21.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:21.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:22.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:22.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:22.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:22.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:22.723 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:32:22.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:22.720971+0000) 2019-10-18T01:32:23.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:23.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:23.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:23.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:23.232 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:32:23.227+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:23.230198+0000) 2019-10-18T01:32:24.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:32:24.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:24.114942+0000) 2019-10-18T01:32:24.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:24.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:24.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:24.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:25.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:25.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:25.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:25.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:25.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:25.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:26.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:26.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:26.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:26.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:27.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:27.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:27.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:27.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:28.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:28.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:28.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:28.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:28.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:32:28.656+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:28.661052+0000) 2019-10-18T01:32:29.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:29.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:29.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:29.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:30.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:30.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:30.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:30.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:30.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:30.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:31.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:31.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:31.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:31.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:32.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:32.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:32.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:32.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:32.725 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:32:32.715+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:32.721187+0000) 2019-10-18T01:32:33.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:32:33.227+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:33.230395+0000) 2019-10-18T01:32:33.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:33.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:33.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:33.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:34.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:32:34.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:34.115042+0000) 2019-10-18T01:32:34.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:34.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:34.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:34.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:35.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:35.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:35.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:35.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:35.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:35.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:36.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:36.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:36.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:36.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:37.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:37.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:37.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:37.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:38.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:38.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:38.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:38.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:38.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:32:38.656+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:38.661155+0000) 2019-10-18T01:32:39.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:39.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:39.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:39.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:40.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:40.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:40.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:40.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:40.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:40.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:41.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:41.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:41.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:41.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:42.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:42.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:42.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:42.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:42.724 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:32:42.714+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:42.721400+0000) 2019-10-18T01:32:43.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:32:43.226+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:43.230544+0000) 2019-10-18T01:32:43.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:43.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:43.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:43.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:44.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:32:44.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:44.115143+0000) 2019-10-18T01:32:44.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:44.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:44.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:44.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:45.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:45.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:45.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:45.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:45.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:45.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:46.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:46.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:46.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:46.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:47.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:47.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:47.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:47.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:48.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:48.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:48.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:48.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:48.663 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:32:48.656+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:48.661253+0000) 2019-10-18T01:32:49.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:49.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:49.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:49.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:50.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:50.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:50.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:50.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:50.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:50.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:51.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:51.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:51.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:51.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:52.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:52.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:52.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:52.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:52.724 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:32:52.714+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:52.721610+0000) 2019-10-18T01:32:53.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:32:53.226+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:53.230728+0000) 2019-10-18T01:32:53.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:53.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:53.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:53.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:54.117 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:32:54.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:54.115243+0000) 2019-10-18T01:32:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:54.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:54.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:55.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:55.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:55.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:55.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:55.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:32:55.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:32:56.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:56.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:56.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:56.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:57.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:57.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:57.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:57.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:58.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:58.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:58.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:58.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:32:58.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:32:58.656+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:32:58.661352+0000) 2019-10-18T01:32:59.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:59.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:32:59.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:32:59.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:00.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:00.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:00.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:00.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:00.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:00.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:01.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:01.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:01.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:01.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:02.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:02.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:02.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:02.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:02.724 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:33:02.714+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:02.721791+0000) 2019-10-18T01:33:03.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:03.226+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:03.230908+0000) 2019-10-18T01:33:03.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:03.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:03.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:03.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:04.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:33:04.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:04.115338+0000) 2019-10-18T01:33:04.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:04.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:04.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:04.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:05.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:05.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:05.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:05.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:05.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:05.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:06.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:06.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:06.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:06.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:07.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:07.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:07.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:07.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:08.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:08.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:08.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:08.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:08.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:33:08.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:08.661454+0000) 2019-10-18T01:33:09.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:09.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:09.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:09.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:10.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:10.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:10.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:10.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:10.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:10.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:11.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:11.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:11.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:11.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:12.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:12.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:12.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:12.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:12.724 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:33:12.714+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:12.721974+0000) 2019-10-18T01:33:13.233 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:13.226+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:13.231099+0000) 2019-10-18T01:33:13.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:13.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:13.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:13.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:14.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:33:14.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:14.115436+0000) 2019-10-18T01:33:14.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:14.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:14.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:14.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:15.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:15.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:15.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:15.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:15.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:15.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:16.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:16.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:16.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:16.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:17.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:17.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:17.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:17.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:18.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:18.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:18.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:18.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:18.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:33:18.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:18.661549+0000) 2019-10-18T01:33:19.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:19.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:19.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:19.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:20.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:20.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:20.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:20.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:20.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:20.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:21.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:21.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:21.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:21.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:22.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:22.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:22.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:22.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:22.725 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:33:22.718+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:22.722204+0000) 2019-10-18T01:33:23.234 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:23.226+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:23.231239+0000) 2019-10-18T01:33:23.234 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:23.226+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:23.232235+0000) 2019-10-18T01:33:23.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:23.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:23.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:23.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:24.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:33:24.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:24.115540+0000) 2019-10-18T01:33:24.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:24.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:24.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:24.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:25.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:25.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:25.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:25.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:25.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:25.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:26.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:26.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:26.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:26.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:27.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:27.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:27.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:27.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:27.734 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:27.726+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:27.731652+0000) 2019-10-18T01:33:28.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:28.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:28.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:28.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:28.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:33:28.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:28.661646+0000) 2019-10-18T01:33:29.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:29.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:29.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:29.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:30.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:30.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:30.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:30.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:30.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:30.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:31.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:31.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:31.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:31.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:32.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:32.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:32.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:32.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:32.725 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:33:32.718+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:32.722383+0000) 2019-10-18T01:33:33.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:33.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:33.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:33.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:34.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:33:34.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:34.115635+0000) 2019-10-18T01:33:34.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:34.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:34.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:34.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:35.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:35.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:35.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:35.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:35.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:35.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:36.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:36.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:36.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:36.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:37.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:37.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:37.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:37.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:37.734 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:37.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:37.731834+0000) 2019-10-18T01:33:38.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:38.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:38.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:38.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:38.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:33:38.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:38.661744+0000) 2019-10-18T01:33:39.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:39.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:39.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:39.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:40.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:40.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:40.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:40.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:40.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:40.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:41.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:41.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:41.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:41.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:42.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:42.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:42.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:42.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:42.725 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:33:42.722+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:42.722534+0000) 2019-10-18T01:33:43.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:43.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:43.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:43.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:44.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:33:44.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:44.115734+0000) 2019-10-18T01:33:44.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:44.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:44.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:44.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:45.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:45.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:45.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:45.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:45.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:45.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:46.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:46.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:46.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:46.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:47.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:47.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:47.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:47.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:47.734 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:47.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:47.732014+0000) 2019-10-18T01:33:48.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:48.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:48.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:48.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:48.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:33:48.658+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:48.661841+0000) 2019-10-18T01:33:49.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:49.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:49.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:49.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:50.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:50.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:50.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:50.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:50.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:50.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:51.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:51.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:51.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:51.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:52.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:52.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:52.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:52.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:52.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:33:52.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:52.722743+0000) 2019-10-18T01:33:53.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:53.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:53.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:53.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:54.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:33:54.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:54.115836+0000) 2019-10-18T01:33:54.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:54.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:54.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:54.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:55.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:55.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:55.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:55.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:55.551 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:33:55.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:33:56.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:56.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:56.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:56.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:57.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:57.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:57.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:57.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:57.735 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:33:57.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:57.732208+0000) 2019-10-18T01:33:58.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:58.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:58.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:58.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:33:58.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:33:58.658+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:33:58.661944+0000) 2019-10-18T01:33:59.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:59.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:33:59.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:33:59.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:00.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:00.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:00.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:00.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:00.551 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:00.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:01.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:01.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:01.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:01.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:02.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:02.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:02.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:02.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:02.725 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:34:02.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:02.722945+0000) 2019-10-18T01:34:03.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:03.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:03.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:03.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:04.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:34:04.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:04.115931+0000) 2019-10-18T01:34:04.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:04.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:04.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:04.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:05.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:05.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:05.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:05.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:05.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:05.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:06.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:06.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:06.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:06.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:07.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:07.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:07.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:07.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:07.735 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:34:07.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:07.732355+0000) 2019-10-18T01:34:08.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:08.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:08.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:08.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:08.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:34:08.658+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:08.662038+0000) 2019-10-18T01:34:09.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:09.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:09.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:09.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:10.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:10.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:10.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:10.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:10.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:10.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:11.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:11.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:11.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:11.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:12.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:12.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:12.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:12.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:12.725 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:34:12.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:12.723127+0000) 2019-10-18T01:34:13.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:13.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:13.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:13.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:14.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:34:14.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:14.116028+0000) 2019-10-18T01:34:14.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:14.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:14.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:14.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:15.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:15.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:15.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:15.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:15.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:16.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:16.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:16.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:16.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:17.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:17.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:17.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:17.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:17.735 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:34:17.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:17.732535+0000) 2019-10-18T01:34:18.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:18.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:18.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:18.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:18.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:34:18.657+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:18.662132+0000) 2019-10-18T01:34:19.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:19.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:19.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:19.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:20.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:20.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:20.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:20.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:20.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:20.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:21.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:21.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:21.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:21.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:22.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:22.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:22.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:22.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:22.726 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:34:22.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:22.723329+0000) 2019-10-18T01:34:23.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:23.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:23.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:23.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:24.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:34:24.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:24.116129+0000) 2019-10-18T01:34:24.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:24.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:24.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:24.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:25.556 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:25.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:25.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:25.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:25.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:25.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:26.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:26.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:26.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:26.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:27.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:27.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:27.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:27.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:27.735 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:34:27.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:27.732718+0000) 2019-10-18T01:34:28.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:34:28.657+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:28.662228+0000) 2019-10-18T01:34:28.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:28.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:28.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:28.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:29.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:29.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:29.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:29.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:30.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:30.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:30.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:30.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:30.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:30.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:31.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:31.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:31.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:31.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:32.726 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:34:32.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:32.723513+0000) 2019-10-18T01:34:32.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:32.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:32.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:32.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:33.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:33.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:33.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:33.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:34.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:34:34.109+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:34.116224+0000) 2019-10-18T01:34:34.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:34.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:34.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:34.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:35.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:35.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:35.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:35.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:35.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:35.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:36.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:36.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:36.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:36.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:37.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:37.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:37.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:37.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:37.735 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:34:37.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:37.732909+0000) 2019-10-18T01:34:38.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:38.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:38.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:38.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:38.664 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:34:38.661+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:38.662326+0000) 2019-10-18T01:34:39.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:39.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:39.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:39.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:40.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:40.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:40.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:40.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:40.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:40.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:41.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:41.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:41.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:41.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:42.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:42.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:42.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:42.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:42.726 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:34:42.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:42.723693+0000) 2019-10-18T01:34:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:43.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:43.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:44.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:34:44.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:44.116321+0000) 2019-10-18T01:34:44.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:44.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:44.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:44.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:45.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:45.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:45.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:45.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:45.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:45.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:46.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:46.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:46.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:46.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:47.735 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:34:47.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:47.733093+0000) 2019-10-18T01:34:47.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:47.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:47.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:47.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:48.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:34:48.661+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:48.662424+0000) 2019-10-18T01:34:48.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:48.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:48.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:48.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:49.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:49.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:49.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:49.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:50.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:50.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:50.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:50.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:50.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:50.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:51.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:51.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:51.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:51.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:52.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:52.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:52.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:52.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:52.726 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:34:52.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:52.723875+0000) 2019-10-18T01:34:53.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:53.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:53.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:53.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:54.118 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:34:54.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:54.116419+0000) 2019-10-18T01:34:54.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:54.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:54.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:54.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:55.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:34:55.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:34:55.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:55.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:55.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:55.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:56.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:56.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:56.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:56.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:57.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:57.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:57.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:57.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:57.736 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:34:57.729+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:57.733274+0000) 2019-10-18T01:34:58.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:58.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:58.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:58.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:34:58.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:34:58.660+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:34:58.662522+0000) 2019-10-18T01:34:59.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:59.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:34:59.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:34:59.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:00.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:00.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:00.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:00.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:00.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:00.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:01.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:01.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:01.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:01.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:02.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:02.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:02.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:02.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:02.726 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:35:02.721+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:02.724083+0000) 2019-10-18T01:35:03.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:03.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:03.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:03.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:04.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:35:04.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:04.116521+0000) 2019-10-18T01:35:04.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:04.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:04.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:04.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:05.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:05.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:05.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:05.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:05.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:05.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:06.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:06.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:06.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:06.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:07.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:07.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:07.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:07.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:07.736 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:35:07.728+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:07.733456+0000) 2019-10-18T01:35:08.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:08.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:08.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:08.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:08.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:35:08.660+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:08.662627+0000) 2019-10-18T01:35:09.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:09.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:09.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:09.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:10.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:10.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:10.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:10.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:10.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:10.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:11.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:11.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:11.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:11.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:12.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:12.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:12.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:12.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:12.727 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:35:12.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:12.724271+0000) 2019-10-18T01:35:13.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:13.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:13.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:13.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:14.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:35:14.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:14.116621+0000) 2019-10-18T01:35:14.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:14.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:14.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:14.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:15.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:15.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:15.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:15.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:15.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:15.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:16.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:16.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:16.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:16.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:17.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:17.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:17.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:17.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:17.736 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:35:17.728+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:17.733637+0000) 2019-10-18T01:35:18.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:18.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:18.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:18.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:18.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:35:18.660+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:18.662725+0000) 2019-10-18T01:35:19.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:19.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:19.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:19.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:20.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:20.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:20.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:20.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:20.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:20.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:21.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:21.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:21.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:21.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:22.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:22.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:22.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:35:22.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:22.724462+0000) 2019-10-18T01:35:23.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:23.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:23.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:23.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:24.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:35:24.112+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:24.116721+0000) 2019-10-18T01:35:24.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:24.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:24.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:24.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:25.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:25.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:25.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:25.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:25.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:25.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:26.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:26.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:26.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:26.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:27.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:27.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:27.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:27.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:27.736 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:35:27.728+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:27.733822+0000) 2019-10-18T01:35:28.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:28.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:28.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:28.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:28.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:35:28.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:28.662823+0000) 2019-10-18T01:35:29.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:29.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:29.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:29.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:30.567 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:30.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:30.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:30.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:30.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:30.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:31.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:31.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:31.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:31.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:32.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:32.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:32.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:32.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:32.727 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:35:32.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:32.724652+0000) 2019-10-18T01:35:33.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:33.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:33.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:33.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:34.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:35:34.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:34.116831+0000) 2019-10-18T01:35:34.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:34.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:34.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:34.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:35.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:35.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:35.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:35.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:35.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:35.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:36.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:36.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:36.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:36.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:37.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:37.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:37.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:37.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:37.736 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:35:37.728+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:37.734010+0000) 2019-10-18T01:35:38.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:38.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:38.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:38.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:38.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:35:38.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:38.662929+0000) 2019-10-18T01:35:39.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:39.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:39.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:39.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:40.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:40.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:40.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:40.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:40.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:40.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:41.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:41.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:41.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:41.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:42.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:42.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:42.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:42.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:42.729 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:35:42.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:42.724841+0000) 2019-10-18T01:35:43.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:43.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:43.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:43.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:44.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:35:44.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:44.116933+0000) 2019-10-18T01:35:44.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:44.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:25:15.415208+0000 (oldest deadline 2019-10-18T01:25:35.415208+0000) 2019-10-18T01:35:44.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:44.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:45.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:45.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:45.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:45.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:46.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:46.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:47.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:47.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:47.737 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:35:47.732+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:47.734224+0000) 2019-10-18T01:35:48.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:48.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:48.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:35:48.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:48.663029+0000) 2019-10-18T01:35:49.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:49.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:50.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:50.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:50.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:50.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:51.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:51.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:52.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:52.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:52.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:35:52.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:52.725054+0000) 2019-10-18T01:35:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:53.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:54.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:35:54.111+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:54.117030+0000) 2019-10-18T01:35:54.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:54.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:55.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:55.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:35:55.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:35:56.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:56.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:57.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:57.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:57.737 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:35:57.732+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:57.734411+0000) 2019-10-18T01:35:58.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:58.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:35:58.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:35:58.659+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:35:58.663129+0000) 2019-10-18T01:35:59.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:35:59.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:00.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:00.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:00.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:00.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:01.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:01.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:02.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:02.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:02.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:02.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:02.725245+0000) 2019-10-18T01:36:03.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:03.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:04.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:36:04.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:04.117129+0000) 2019-10-18T01:36:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:04.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:05.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:05.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:05.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:05.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:06.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:06.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:07.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:07.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:07.737 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:36:07.732+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:07.734619+0000) 2019-10-18T01:36:08.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:08.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:08.665 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:08.658+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:08.663226+0000) 2019-10-18T01:36:09.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:09.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:09.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:09.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:10.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:10.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:10.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:10.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:10.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:10.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:11.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:11.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:11.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:11.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:12.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:12.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:12.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:12.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:12.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:12.720+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:12.725426+0000) 2019-10-18T01:36:13.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:13.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:13.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:13.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:14.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:36:14.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:14.117228+0000) 2019-10-18T01:36:14.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:14.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:14.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:14.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:15.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:15.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:15.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:15.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:15.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:16.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:16.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:16.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:16.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:17.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:17.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:17.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:17.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:17.737 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:36:17.732+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:17.734833+0000) 2019-10-18T01:36:18.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:18.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:18.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:18.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:18.666 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:18.658+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:18.663321+0000) 2019-10-18T01:36:18.666 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:18.658+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:18.664299+0000) 2019-10-18T01:36:19.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:19.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:19.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:19.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:20.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:20.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:20.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:20.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:20.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:20.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:21.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:21.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:21.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:21.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:22.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:22.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:22.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:22.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:22.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:22.719+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:22.725606+0000) 2019-10-18T01:36:23.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:23.158+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:23.163523+0000) 2019-10-18T01:36:23.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:23.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:23.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:23.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:24.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:36:24.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:24.117329+0000) 2019-10-18T01:36:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:24.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:24.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:25.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:25.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:25.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:25.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:25.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:25.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:26.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:26.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:26.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:26.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:27.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:27.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:27.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:27.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:27.737 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:36:27.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:27.735052+0000) 2019-10-18T01:36:28.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:28.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:28.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:28.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:29.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:29.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:29.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:29.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:30.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:30.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:30.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:30.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:30.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:30.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:31.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:31.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:31.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:31.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:32.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:32.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:32.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:32.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:32.728 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:32.719+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:32.725771+0000) 2019-10-18T01:36:33.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:33.158+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:33.163628+0000) 2019-10-18T01:36:33.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:33.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:33.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:33.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:34.119 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:36:34.110+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:34.117425+0000) 2019-10-18T01:36:34.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:34.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:34.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:34.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:35.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:35.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:35.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:35.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:35.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:35.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:36.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:36.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:36.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:36.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:37.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:37.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:37.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:37.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:37.738 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:36:37.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:37.735233+0000) 2019-10-18T01:36:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:38.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:38.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:38.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:39.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:39.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:39.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:39.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:40.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:40.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:40.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:40.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:40.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:40.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:41.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:41.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:41.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:41.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:42.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:42.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:42.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:42.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:42.729 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:42.719+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:42.725961+0000) 2019-10-18T01:36:42.729 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:42.723+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:42.726863+0000) 2019-10-18T01:36:43.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:43.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:43.163733+0000) 2019-10-18T01:36:43.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:43.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:43.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:43.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:44.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:36:44.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:44.117532+0000) 2019-10-18T01:36:44.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:44.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:44.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:44.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:45.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:45.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:45.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:45.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:45.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:45.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:46.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:46.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:46.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:46.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:47.229 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:47.223+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:47.226445+0000) 2019-10-18T01:36:47.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:47.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:47.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:47.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:47.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:36:47.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:47.735416+0000) 2019-10-18T01:36:48.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:48.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:48.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:48.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:49.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:49.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:49.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:49.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:50.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:50.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:50.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:50.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:50.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:50.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:51.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:51.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:51.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:51.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:52.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:52.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:52.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:52.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:53.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:36:53.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:53.163840+0000) 2019-10-18T01:36:53.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:53.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:53.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:53.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:54.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:36:54.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:54.117628+0000) 2019-10-18T01:36:54.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:54.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:54.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:54.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:55.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:55.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:55.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:55.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:55.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:36:55.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:36:56.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:56.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:56.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:56.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:57.229 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:36:57.223+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:57.226646+0000) 2019-10-18T01:36:57.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:57.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:57.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:57.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:57.738 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:36:57.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:36:57.735602+0000) 2019-10-18T01:36:58.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:58.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:58.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:58.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:36:59.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:59.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:36:59.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:36:59.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:00.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:00.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:00.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:00.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:00.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:00.577+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:01.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:01.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:01.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:01.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:02.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:02.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:02.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:02.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:03.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:37:03.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:03.163945+0000) 2019-10-18T01:37:03.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:03.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:03.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:03.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:04.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:37:04.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:04.117722+0000) 2019-10-18T01:37:04.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:04.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:04.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:04.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:05.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:05.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:05.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:05.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:05.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:05.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:06.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:06.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:06.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:06.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:07.229 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:37:07.223+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:07.226830+0000) 2019-10-18T01:37:07.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:07.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:07.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:07.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:07.738 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:37:07.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:07.735811+0000) 2019-10-18T01:37:08.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:08.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:08.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:08.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:09.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:09.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:09.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:09.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:10.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:10.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:10.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:10.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:10.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:10.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:11.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:11.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:11.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:11.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:12.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:12.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:12.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:12.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:13.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:37:13.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:13.164037+0000) 2019-10-18T01:37:13.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:13.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:13.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:13.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:14.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:37:14.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:14.117818+0000) 2019-10-18T01:37:14.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:14.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:14.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:14.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:15.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:15.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:15.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:15.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:15.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:16.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:16.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:16.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:16.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:17.229 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:37:17.223+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:17.227011+0000) 2019-10-18T01:37:17.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:17.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:17.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:17.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:17.738 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:37:17.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:17.736020+0000) 2019-10-18T01:37:18.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:18.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:18.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:18.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:19.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:19.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:19.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:19.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:20.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:20.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:20.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:20.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:20.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:20.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:21.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:21.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:21.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:21.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:22.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:22.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:22.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:22.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:23.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:37:23.160+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:23.164129+0000) 2019-10-18T01:37:23.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:23.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:23.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:23.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:24.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:37:24.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:24.117915+0000) 2019-10-18T01:37:24.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:24.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:24.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:24.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:25.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:25.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:25.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:25.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:25.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:25.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:26.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:26.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:26.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:26.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:27.229 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:37:27.223+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:27.227193+0000) 2019-10-18T01:37:27.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:27.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:27.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:27.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:27.739 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:37:27.731+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:27.736227+0000) 2019-10-18T01:37:28.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:28.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:28.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:28.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:29.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:29.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:29.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:29.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:30.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:30.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:30.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:30.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:30.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:30.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:31.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:31.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:31.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:31.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:32.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:32.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:32.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:32.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:33.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:37:33.160+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:33.164229+0000) 2019-10-18T01:37:33.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:33.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:33.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:33.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:34.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:37:34.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:34.118010+0000) 2019-10-18T01:37:34.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:34.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:34.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:34.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:35.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:35.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:35.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:35.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:35.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:35.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:36.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:36.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:36.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:36.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:37.230 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:37:37.222+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:37.227376+0000) 2019-10-18T01:37:37.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:37.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:37.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:37.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:37.739 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:37:37.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:37.736436+0000) 2019-10-18T01:37:38.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:38.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:38.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:38.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:39.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:39.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:39.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:39.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:40.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:40.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:40.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:40.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:40.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:40.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:41.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:41.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:41.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:41.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:42.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:42.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:42.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:42.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:43.166 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:37:43.160+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:43.164328+0000) 2019-10-18T01:37:43.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:43.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:43.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:43.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:44.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:37:44.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:44.118108+0000) 2019-10-18T01:37:44.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:44.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:44.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:44.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:45.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:45.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:45.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:45.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:45.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:45.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:46.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:46.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:46.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:46.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:47.230 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:37:47.222+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:47.227559+0000) 2019-10-18T01:37:47.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:47.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:47.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:47.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:47.739 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:37:47.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:47.736620+0000) 2019-10-18T01:37:48.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:48.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:48.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:48.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:49.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:49.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:49.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:49.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:50.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:50.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:50.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:50.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:50.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:50.588+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:51.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:51.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:51.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:51.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:52.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:52.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:52.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:52.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:53.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:37:53.159+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:53.164435+0000) 2019-10-18T01:37:53.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:53.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:53.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:53.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:54.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:37:54.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:54.118204+0000) 2019-10-18T01:37:54.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:54.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:54.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:54.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:55.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:55.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:55.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:55.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:55.593 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:37:55.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:37:56.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:56.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:56.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:56.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:57.230 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:37:57.222+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:57.227755+0000) 2019-10-18T01:37:57.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:57.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:57.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:57.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:57.739 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:37:57.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:37:57.736802+0000) 2019-10-18T01:37:58.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:58.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:58.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:58.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:37:59.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:59.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:37:59.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:37:59.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:00.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:00.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:00.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:00.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:00.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:00.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:01.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:01.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:01.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:01.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:02.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:02.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:02.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:02.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:03.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:38:03.159+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:03.164536+0000) 2019-10-18T01:38:03.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:03.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:03.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:03.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:04.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:38:04.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:04.118307+0000) 2019-10-18T01:38:04.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:04.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:04.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:04.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:05.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:05.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:05.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:05.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:05.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:05.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:06.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:06.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:06.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:06.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:07.230 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:38:07.222+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:07.227940+0000) 2019-10-18T01:38:07.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:07.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:07.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:07.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:07.739 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:38:07.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:07.737008+0000) 2019-10-18T01:38:08.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:08.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:08.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:08.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:09.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:09.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:09.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:09.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:10.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:10.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:10.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:10.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:11.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:11.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:11.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:11.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:12.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:12.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:12.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:12.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:13.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:38:13.159+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:13.164639+0000) 2019-10-18T01:38:13.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:13.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:13.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:13.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:14.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:38:14.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:14.118405+0000) 2019-10-18T01:38:14.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:14.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:14.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:14.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:15.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:15.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:15.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:15.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:15.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:15.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:16.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:16.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:16.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:16.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:17.231 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:38:17.222+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:17.228143+0000) 2019-10-18T01:38:17.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:17.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:17.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:17.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:17.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:38:17.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:17.737192+0000) 2019-10-18T01:38:18.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:18.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:18.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:18.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:19.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:19.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:19.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:19.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:20.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:20.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:20.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:20.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:20.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:20.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:21.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:21.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:21.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:21.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:22.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:22.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:22.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:22.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:23.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:38:23.159+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:23.164744+0000) 2019-10-18T01:38:23.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:23.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:23.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:23.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:24.120 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:38:24.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:24.118503+0000) 2019-10-18T01:38:24.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:24.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:24.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:24.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:25.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:25.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:25.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:25.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:25.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:25.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:26.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:26.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:26.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:26.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:27.231 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:38:27.222+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:27.228322+0000) 2019-10-18T01:38:27.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:27.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:27.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:27.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:27.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:38:27.730+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:27.737384+0000) 2019-10-18T01:38:28.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:28.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:28.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:28.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:29.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:29.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:29.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:29.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:30.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:30.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:30.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:30.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:30.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:30.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:31.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:31.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:31.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:31.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:32.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:32.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:32.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:32.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:33.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:38:33.158+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:33.164860+0000) 2019-10-18T01:38:33.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:33.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:33.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:33.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:34.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:38:34.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:34.118595+0000) 2019-10-18T01:38:34.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:34.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:34.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:34.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:35.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:35.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:35.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:35.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:35.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:35.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:36.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:36.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:36.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:36.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:37.231 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:38:37.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:37.228503+0000) 2019-10-18T01:38:37.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:37.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:37.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:37.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:37.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:38:37.734+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:37.737560+0000) 2019-10-18T01:38:38.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:38.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:38.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:38.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:39.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:39.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:39.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:39.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:40.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:40.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:40.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:40.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:40.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:40.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:41.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:41.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:41.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:41.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:42.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:42.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:42.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:42.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:43.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:38:43.158+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:43.164964+0000) 2019-10-18T01:38:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:43.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:43.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:44.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:38:44.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:44.118695+0000) 2019-10-18T01:38:44.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:44.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:44.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:44.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:45.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:45.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:45.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:45.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:45.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:45.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:46.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:46.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:46.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:46.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:47.231 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:38:47.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:47.228719+0000) 2019-10-18T01:38:47.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:47.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:47.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:47.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:47.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:38:47.733+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:47.737750+0000) 2019-10-18T01:38:48.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:48.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:48.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:48.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:49.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:49.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:49.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:49.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:50.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:50.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:50.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:50.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:50.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:50.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:51.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:51.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:51.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:51.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:52.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:52.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:52.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:52.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:53.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:38:53.158+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:53.165070+0000) 2019-10-18T01:38:53.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:53.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:53.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:53.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:54.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:38:54.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:54.118795+0000) 2019-10-18T01:38:54.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:54.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:54.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:54.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:55.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:55.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:55.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:55.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:55.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:38:55.598+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:38:56.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:56.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:56.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:56.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:57.231 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:38:57.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:57.228902+0000) 2019-10-18T01:38:57.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:57.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:57.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:57.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:57.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:38:57.733+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:38:57.737942+0000) 2019-10-18T01:38:58.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:58.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:58.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:58.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:38:59.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:59.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:38:59.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:38:59.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:00.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:00.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:00.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:00.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:00.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:00.598+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:01.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:01.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:01.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:01.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:02.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:02.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:02.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:02.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:03.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:39:03.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:03.165176+0000) 2019-10-18T01:39:03.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:03.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:03.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:03.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:04.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:39:04.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:04.118892+0000) 2019-10-18T01:39:04.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:04.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:04.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:04.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:05.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:05.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:05.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:05.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:05.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:05.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:06.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:06.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:06.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:06.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:07.231 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:39:07.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:07.229090+0000) 2019-10-18T01:39:07.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:07.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:07.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:07.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:07.740 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:39:07.733+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:07.738153+0000) 2019-10-18T01:39:08.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:08.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:08.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:08.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:09.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:09.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:09.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:09.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:10.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:10.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:10.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:10.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:10.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:10.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:11.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:11.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:11.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:11.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:12.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:12.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:12.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:12.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:13.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:39:13.157+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:13.165286+0000) 2019-10-18T01:39:13.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:13.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:13.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:13.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:14.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:39:14.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:14.118992+0000) 2019-10-18T01:39:14.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:14.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:14.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:14.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:15.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:15.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:15.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:15.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:15.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:15.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:16.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:16.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:16.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:16.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:17.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:39:17.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:17.229278+0000) 2019-10-18T01:39:17.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:17.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:17.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:17.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:17.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:39:17.737+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:17.738382+0000) 2019-10-18T01:39:18.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:18.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:18.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:18.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:19.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:19.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:19.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:19.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:20.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:20.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:20.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:20.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:20.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:20.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:21.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:21.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:21.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:21.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:22.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:22.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:22.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:22.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:23.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:39:23.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:23.165397+0000) 2019-10-18T01:39:23.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:23.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:23.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:23.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:24.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:39:24.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:24.119088+0000) 2019-10-18T01:39:24.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:24.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:24.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:24.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:25.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:25.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:25.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:25.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:25.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:25.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:26.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:26.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:26.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:26.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:27.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:27.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:27.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:27.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:27.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:39:27.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:27.229459+0000) 2019-10-18T01:39:27.745 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:39:27.737+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:27.738551+0000) 2019-10-18T01:39:28.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:28.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:28.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:28.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:29.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:29.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:29.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:29.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:30.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:30.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:30.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:30.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:30.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:30.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:31.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:31.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:31.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:31.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:32.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:32.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:32.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:32.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:33.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:39:33.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:33.165499+0000) 2019-10-18T01:39:33.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:33.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:33.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:33.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:34.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:39:34.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:34.119186+0000) 2019-10-18T01:39:34.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:34.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:34.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:34.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:35.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:35.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:35.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:35.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:35.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:35.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:36.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:36.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:36.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:36.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:37.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:39:37.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:37.229658+0000) 2019-10-18T01:39:37.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:37.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:37.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:37.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:37.741 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:39:37.737+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:37.738762+0000) 2019-10-18T01:39:38.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:38.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:38.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:38.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:39.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:39.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:39.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:39.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:40.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:40.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:40.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:40.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:40.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:41.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:41.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:41.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:41.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:42.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:42.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:42.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:42.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:43.167 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:39:43.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:43.165602+0000) 2019-10-18T01:39:43.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:43.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:43.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:43.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:44.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:39:44.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:44.119282+0000) 2019-10-18T01:39:44.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:44.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:44.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:44.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:45.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:45.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:45.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:45.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:45.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:45.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:46.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:46.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:46.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:46.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:47.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:39:47.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:47.229843+0000) 2019-10-18T01:39:47.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:47.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:47.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:47.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:47.741 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:39:47.737+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:47.738942+0000) 2019-10-18T01:39:48.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:48.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:48.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:48.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:49.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:49.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:49.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:49.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:50.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:50.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:50.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:50.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:50.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:50.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:51.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:51.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:51.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:51.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:52.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:52.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:52.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:52.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:53.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:39:53.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:53.165707+0000) 2019-10-18T01:39:53.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:53.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:53.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:53.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:54.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:39:54.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:54.119375+0000) 2019-10-18T01:39:54.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:54.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:54.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:54.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:55.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:55.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:55.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:55.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:55.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:39:55.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:39:56.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:56.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:56.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:56.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:57.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:57.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:57.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:57.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:57.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:39:57.225+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:57.230019+0000) 2019-10-18T01:39:57.741 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:39:57.737+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:39:57.739154+0000) 2019-10-18T01:39:58.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:58.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:58.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:58.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:39:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:59.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:39:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:39:59.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:00.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:00.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:00.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:00.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:00.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:00.612+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:01.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:01.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:01.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:01.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:02.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:02.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:02.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:02.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:03.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:03.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:03.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:03.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:03.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:40:03.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:03.165811+0000) 2019-10-18T01:40:04.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:40:04.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:04.119471+0000) 2019-10-18T01:40:04.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:04.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:04.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:04.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:05.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:05.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:05.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:05.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:05.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:05.612+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:06.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:06.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:06.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:06.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:07.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:07.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:07.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:07.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:07.232 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:40:07.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:07.230207+0000) 2019-10-18T01:40:07.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:40:07.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:07.739331+0000) 2019-10-18T01:40:08.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:08.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:08.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:08.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:09.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:09.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:09.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:09.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:10.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:10.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:10.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:10.612+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:11.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:11.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:11.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:11.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:12.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:12.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:12.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:12.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:13.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:40:13.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:13.165919+0000) 2019-10-18T01:40:13.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:13.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:13.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:13.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:14.121 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:40:14.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:14.119575+0000) 2019-10-18T01:40:14.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:14.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:14.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:14.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:15.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:15.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:15.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:15.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:15.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:15.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:16.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:16.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:16.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:16.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:17.233 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:40:17.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:17.230372+0000) 2019-10-18T01:40:17.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:17.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:17.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:17.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:17.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:40:17.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:17.739517+0000) 2019-10-18T01:40:18.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:18.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:18.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:18.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:19.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:19.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:19.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:19.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:20.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:20.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:20.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:20.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:20.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:20.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:21.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:21.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:21.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:21.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:22.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:22.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:22.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:22.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:23.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:40:23.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:23.166024+0000) 2019-10-18T01:40:23.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:23.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:23.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:23.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:24.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:40:24.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:24.119675+0000) 2019-10-18T01:40:24.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:24.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:24.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:24.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:25.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:25.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:25.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:25.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:25.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:25.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:26.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:26.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:26.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:26.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:27.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:27.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:27.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:27.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:27.233 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:40:27.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:27.230534+0000) 2019-10-18T01:40:27.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:40:27.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:27.739731+0000) 2019-10-18T01:40:28.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:28.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:28.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:28.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:29.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:29.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:29.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:29.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:30.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:30.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:30.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:30.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:30.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:30.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:31.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:31.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:31.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:31.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:32.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:32.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:32.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:32.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:33.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:40:33.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:33.166127+0000) 2019-10-18T01:40:33.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:33.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:33.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:33.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:34.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:40:34.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:34.119772+0000) 2019-10-18T01:40:34.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:34.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:34.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:34.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:35.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:35.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:35.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:35.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:35.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:35.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:36.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:36.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:36.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:36.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:37.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:37.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:37.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:37.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:37.233 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:40:37.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:37.230701+0000) 2019-10-18T01:40:37.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:40:37.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:37.739941+0000) 2019-10-18T01:40:38.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:38.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:38.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:38.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:39.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:39.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:39.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:39.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:40.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:40.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:40.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:40.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:40.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:40.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:41.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:41.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:41.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:41.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:42.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:42.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:42.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:42.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:43.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:43.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:43.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:43.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:43.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:40:43.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:43.166231+0000) 2019-10-18T01:40:44.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:44.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:44.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:44.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:44.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:40:44.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:44.119871+0000) 2019-10-18T01:40:45.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:45.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:45.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:45.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:45.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:45.619+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:46.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:46.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:46.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:46.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:47.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:47.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:47.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:47.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:47.233 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:40:47.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:47.230890+0000) 2019-10-18T01:40:47.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:40:47.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:47.740158+0000) 2019-10-18T01:40:48.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:48.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:48.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:48.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:48.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:48.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:48.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:48.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:49.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:49.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:49.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:49.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:50.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:50.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:50.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:50.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:50.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:50.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:52.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:52.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:52.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:52.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:53.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:53.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:53.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:53.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:53.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:40:53.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:53.166336+0000) 2019-10-18T01:40:54.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:54.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:54.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:54.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:54.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:40:54.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:54.119973+0000) 2019-10-18T01:40:55.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:55.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:55.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:55.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:55.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:40:55.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:40:56.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:56.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:56.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:56.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:57.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:57.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:57.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:57.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:57.233 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:40:57.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:57.231089+0000) 2019-10-18T01:40:57.742 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:40:57.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:40:57.740345+0000) 2019-10-18T01:40:58.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:58.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:58.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:58.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:40:59.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:59.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:40:59.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:40:59.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:00.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:00.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:00.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:00.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:00.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:00.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:01.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:01.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:01.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:01.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:02.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:02.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:02.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:02.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:03.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:03.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:03.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:03.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:03.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:41:03.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:03.166441+0000) 2019-10-18T01:41:04.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:04.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:04.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:04.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:04.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:41:04.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:04.120075+0000) 2019-10-18T01:41:05.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:05.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:05.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:05.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:05.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:05.622+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:06.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:06.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:06.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:06.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:07.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:07.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:07.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:07.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:07.233 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:41:07.228+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:07.231303+0000) 2019-10-18T01:41:07.743 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:41:07.736+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:07.740567+0000) 2019-10-18T01:41:07.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:07.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:07.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:07.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:08.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:08.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:08.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:08.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:09.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:09.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:09.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:09.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:10.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:10.622+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:10.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:10.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:10.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:10.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:11.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:11.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:11.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:11.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:12.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:12.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:12.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:12.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:13.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:41:13.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:13.166554+0000) 2019-10-18T01:41:13.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:13.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:13.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:13.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:14.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:41:14.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:14.120174+0000) 2019-10-18T01:41:14.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:14.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:14.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:14.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:15.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:15.622+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:15.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:15.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:15.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:15.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:17.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:17.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:17.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:17.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:17.234 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:41:17.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:17.231457+0000) 2019-10-18T01:41:17.743 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:41:17.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:17.740758+0000) 2019-10-18T01:41:18.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:18.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:18.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:18.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:19.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:19.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:19.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:19.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:20.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:20.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:20.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:20.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:20.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:20.622+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:21.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:21.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:21.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:21.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:22.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:22.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:22.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:22.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:23.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:23.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:23.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:23.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:23.168 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:41:23.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:23.166664+0000) 2019-10-18T01:41:24.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:24.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:24.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:24.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:24.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:41:24.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:24.120276+0000) 2019-10-18T01:41:25.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:25.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:25.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:25.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:25.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:25.622+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:26.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:26.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:26.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:26.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:27.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:27.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:27.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:27.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:27.234 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:41:27.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:27.231670+0000) 2019-10-18T01:41:27.743 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:41:27.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:27.740943+0000) 2019-10-18T01:41:28.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:28.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:28.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:28.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:29.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:29.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:29.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:29.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:30.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:30.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:30.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:30.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:30.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:30.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:31.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:31.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:31.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:31.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:32.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:32.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:32.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:32.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:33.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:33.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:33.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:33.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:33.169 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:41:33.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:33.166767+0000) 2019-10-18T01:41:34.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:34.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:34.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:34.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:34.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:41:34.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:34.120373+0000) 2019-10-18T01:41:35.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:35.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:35.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:35.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:35.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:35.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:36.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:36.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:36.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:36.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:37.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:37.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:37.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:37.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:37.234 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:41:37.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:37.231883+0000) 2019-10-18T01:41:37.743 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:41:37.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:37.741122+0000) 2019-10-18T01:41:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:38.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:38.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:39.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:39.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:39.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:39.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:40.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:40.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:40.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:40.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:40.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:40.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:41.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:41.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:41.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:41.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:42.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:42.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:42.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:42.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:43.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:43.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:43.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:43.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:43.169 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:41:43.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:43.166913+0000) 2019-10-18T01:41:44.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:44.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:44.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:44.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:44.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:41:44.113+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:44.120474+0000) 2019-10-18T01:41:45.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:45.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:45.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:45.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:45.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:45.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:46.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:46.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:46.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:46.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:47.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:47.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:47.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:47.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:47.234 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:41:47.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:47.232067+0000) 2019-10-18T01:41:47.744 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:41:47.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:47.741330+0000) 2019-10-18T01:41:48.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:48.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:48.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:48.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:49.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:49.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:49.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:49.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:50.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:50.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:50.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:50.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:51.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:51.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:51.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:51.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:52.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:52.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:52.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:52.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:53.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:53.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:53.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:53.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:53.169 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:41:53.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:53.167021+0000) 2019-10-18T01:41:54.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:54.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:54.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:54.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:54.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:41:54.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:54.120573+0000) 2019-10-18T01:41:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:55.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:55.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:55.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:41:55.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:41:56.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:56.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:56.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:56.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:57.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:57.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:57.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:57.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:57.234 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:41:57.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:57.232265+0000) 2019-10-18T01:41:57.744 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:41:57.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:41:57.741508+0000) 2019-10-18T01:41:58.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:58.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:58.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:58.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:41:59.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:59.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:41:59.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:41:59.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:00.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:00.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:00.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:00.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:00.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:00.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:01.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:01.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:01.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:01.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:02.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:02.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:02.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:02.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:03.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:03.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:03.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:03.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:03.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:42:03.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:03.167125+0000) 2019-10-18T01:42:04.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:04.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:04.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:04.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:04.122 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:42:04.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:04.120678+0000) 2019-10-18T01:42:05.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:05.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:05.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:05.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:05.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:05.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:06.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:06.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:06.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:06.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:07.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:07.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:07.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:07.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:07.234 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:42:07.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:07.232458+0000) 2019-10-18T01:42:07.744 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:42:07.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:07.741718+0000) 2019-10-18T01:42:08.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:08.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:08.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:08.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:09.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:09.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:09.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:09.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:10.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:10.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:10.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:10.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:10.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:10.636+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:11.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:11.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:11.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:11.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:12.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:12.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:12.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:12.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:13.169 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:42:13.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:13.167235+0000) 2019-10-18T01:42:13.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:13.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:13.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:13.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:14.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:42:14.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:14.120777+0000) 2019-10-18T01:42:14.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:14.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:14.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:14.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:15.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:15.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:15.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:15.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:15.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:15.636+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:16.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:16.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:16.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:16.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:17.242 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:42:17.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:17.232648+0000) 2019-10-18T01:42:17.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:17.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:17.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:17.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:17.744 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:42:17.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:17.741897+0000) 2019-10-18T01:42:18.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:18.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:18.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:18.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:19.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:19.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:19.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:19.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:20.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:20.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:20.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:20.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:20.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:20.636+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:21.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:21.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:21.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:21.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:22.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:22.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:22.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:22.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:23.169 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:42:23.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:23.167344+0000) 2019-10-18T01:42:23.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:23.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:23.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:23.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:24.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:42:24.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:24.120878+0000) 2019-10-18T01:42:24.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:24.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:24.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:24.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:25.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:25.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:25.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:25.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:25.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:25.636+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:26.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:26.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:26.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:26.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:27.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:27.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:27.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:27.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:27.235 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:42:27.227+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:27.232855+0000) 2019-10-18T01:42:27.744 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:42:27.735+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:27.742077+0000) 2019-10-18T01:42:28.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:28.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:28.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:28.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:29.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:29.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:29.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:29.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:30.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:30.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:30.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:30.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:30.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:30.636+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:31.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:31.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:31.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:31.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:32.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:32.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:32.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:32.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:33.169 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:42:33.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:33.167438+0000) 2019-10-18T01:42:33.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:33.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:33.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:33.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:34.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:42:34.116+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:34.120979+0000) 2019-10-18T01:42:34.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:34.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:34.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:34.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:35.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:35.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:35.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:35.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:35.644 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:35.640+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:36.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:36.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:36.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:36.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:37.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:37.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:37.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:37.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:37.235 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:42:37.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:37.233042+0000) 2019-10-18T01:42:37.744 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:42:37.738+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:37.742263+0000) 2019-10-18T01:42:38.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:38.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:38.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:38.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:39.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:39.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:39.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:39.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:40.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:40.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:40.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:40.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:40.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:40.639+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:41.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:41.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:41.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:41.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:42.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:42.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:42.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:42.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:43.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:43.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:43.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:43.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:43.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:42:43.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:43.167531+0000) 2019-10-18T01:42:44.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:44.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:44.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:44.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:44.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:42:44.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:44.121080+0000) 2019-10-18T01:42:45.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:45.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:45.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:45.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:45.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:45.639+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:46.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:46.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:46.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:46.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:47.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:47.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:47.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:47.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:47.235 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:42:47.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:47.233215+0000) 2019-10-18T01:42:47.745 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:42:47.738+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:47.742470+0000) 2019-10-18T01:42:48.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:48.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:48.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:48.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:49.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:49.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:49.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:49.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:50.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:50.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:50.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:50.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:50.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:50.639+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:51.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:51.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:51.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:51.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:52.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:52.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:52.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:52.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:53.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:53.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:53.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:53.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:53.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:42:53.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:53.167630+0000) 2019-10-18T01:42:54.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:42:54.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:54.121183+0000) 2019-10-18T01:42:54.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:54.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:54.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:54.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:55.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:55.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:55.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:55.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:55.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:42:55.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:42:56.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:56.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:56.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:56.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:57.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:57.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:57.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:57.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:57.236 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:42:57.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:57.233406+0000) 2019-10-18T01:42:57.745 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:42:57.738+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:42:57.742679+0000) 2019-10-18T01:42:58.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:58.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:58.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:58.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:42:59.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:59.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:42:59.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:42:59.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:00.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:00.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:00.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:00.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:00.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:00.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:01.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:01.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:01.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:01.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:02.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:02.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:02.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:02.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:03.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:43:03.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:03.167724+0000) 2019-10-18T01:43:03.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:03.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:03.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:03.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:04.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:43:04.115+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:04.121284+0000) 2019-10-18T01:43:04.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:04.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:04.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:04.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:05.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:05.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:05.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:05.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:05.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:05.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:06.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:06.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:06.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:06.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:07.236 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:43:07.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:07.233592+0000) 2019-10-18T01:43:07.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:07.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:07.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:07.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:07.745 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:43:07.738+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:07.742865+0000) 2019-10-18T01:43:08.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:08.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:08.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:08.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:09.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:09.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:09.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:09.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:10.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:10.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:10.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:10.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:10.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:10.643+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:11.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:11.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:11.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:11.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:12.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:12.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:12.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:12.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:13.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:43:13.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:13.167821+0000) 2019-10-18T01:43:13.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:13.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:13.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:13.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:14.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:43:14.114+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:14.121382+0000) 2019-10-18T01:43:14.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:14.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:14.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:14.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:15.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:15.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:15.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:15.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:15.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:15.646+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:16.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:16.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:16.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:16.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:17.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:17.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:17.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:17.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:17.236 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:43:17.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:17.233776+0000) 2019-10-18T01:43:17.745 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:43:17.738+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:17.743060+0000) 2019-10-18T01:43:18.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:18.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:18.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:18.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:19.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:19.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:19.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:19.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:20.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:20.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:20.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:20.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:20.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:20.646+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:21.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:21.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:21.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:21.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:22.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:22.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:22.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:22.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:23.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:43:23.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:23.167921+0000) 2019-10-18T01:43:23.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:23.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:23.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:23.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:24.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:43:24.118+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:24.121458+0000) 2019-10-18T01:43:24.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:24.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:24.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:24.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:25.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:25.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:25.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:25.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:25.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:25.646+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:26.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:26.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:26.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:26.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:27.236 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:43:27.226+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:27.233970+0000) 2019-10-18T01:43:27.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:27.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:27.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:27.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:27.745 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:43:27.738+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:27.743253+0000) 2019-10-18T01:43:28.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:28.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:28.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:28.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:29.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:29.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:29.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:29.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:30.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:30.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:30.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:30.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:30.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:30.646+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:31.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:31.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:31.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:31.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:32.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:32.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:32.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:32.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:33.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:43:33.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:33.168018+0000) 2019-10-18T01:43:33.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:33.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:33.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:33.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:34.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:43:34.118+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:34.121555+0000) 2019-10-18T01:43:34.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:34.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:34.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:34.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:35.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:35.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:35.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:35.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:35.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:35.646+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:36.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:36.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:36.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:36.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:37.237 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:43:37.234+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:37.234205+0000) 2019-10-18T01:43:37.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:37.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:37.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:37.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:37.746 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:43:37.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:37.743436+0000) 2019-10-18T01:43:38.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:38.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:38.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:38.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:39.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:39.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:39.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:39.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:40.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:40.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:40.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:40.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:40.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:40.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:41.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:41.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:41.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:41.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:42.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:42.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:42.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:42.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:43.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:43.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:43.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:43.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:43.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:43:43.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:43.168121+0000) 2019-10-18T01:43:44.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:43:44.118+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:44.121652+0000) 2019-10-18T01:43:44.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:44.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:44.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:44.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:45.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:45.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:45.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:45.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:45.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:45.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:46.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:46.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:46.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:46.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:47.238 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:43:47.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:47.234376+0000) 2019-10-18T01:43:47.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:47.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:47.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:47.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:47.746 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:43:47.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:47.743622+0000) 2019-10-18T01:43:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:48.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:48.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:49.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:49.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:49.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:49.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:50.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:50.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:50.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:50.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:50.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:50.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:51.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:51.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:51.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:51.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:52.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:52.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:52.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:52.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:53.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:43:53.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:53.168221+0000) 2019-10-18T01:43:53.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:53.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:53.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:53.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:54.123 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:43:54.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:54.121752+0000) 2019-10-18T01:43:54.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:54.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:54.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:54.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:55.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:55.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:55.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:55.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:55.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:43:55.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:43:56.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:56.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:56.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:56.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:57.237 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:43:57.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:57.234522+0000) 2019-10-18T01:43:57.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:57.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:57.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:57.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:57.746 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:43:57.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:43:57.743807+0000) 2019-10-18T01:43:58.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:58.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:58.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:58.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:43:59.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:59.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:43:59.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:43:59.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:00.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:00.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:00.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:00.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:00.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:00.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:01.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:01.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:01.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:01.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:02.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:02.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:02.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:02.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:03.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:44:03.161+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:03.168318+0000) 2019-10-18T01:44:03.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:03.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:03.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:03.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:04.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:44:04.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:04.121854+0000) 2019-10-18T01:44:04.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:04.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:04.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:04.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:05.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:05.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:05.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:05.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:05.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:05.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:06.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:06.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:06.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:06.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:07.237 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:07.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:07.234705+0000) 2019-10-18T01:44:07.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:07.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:07.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:07.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:07.746 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:44:07.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:07.744009+0000) 2019-10-18T01:44:08.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:08.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:08.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:08.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:09.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:09.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:09.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:09.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:10.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:10.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:10.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:10.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:10.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:10.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:11.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:11.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:11.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:11.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:12.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:12.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:12.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:12.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:13.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:13.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:13.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:13.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:13.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:44:13.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:13.168424+0000) 2019-10-18T01:44:14.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:44:14.121+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:14.121961+0000) 2019-10-18T01:44:14.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:14.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:14.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:14.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:15.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:15.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:15.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:15.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:15.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:15.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:16.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:16.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:16.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:16.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:17.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:17.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:17.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:17.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:17.237 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:17.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:17.234896+0000) 2019-10-18T01:44:17.746 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:44:17.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:17.744188+0000) 2019-10-18T01:44:18.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:18.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:18.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:18.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:19.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:19.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:19.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:19.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:20.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:20.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:20.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:20.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:20.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:20.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:21.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:21.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:21.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:22.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:22.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:22.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:22.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:23.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:23.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:23.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:23.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:23.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:44:23.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:23.168523+0000) 2019-10-18T01:44:24.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:44:24.121+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:24.122063+0000) 2019-10-18T01:44:24.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:24.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:24.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:24.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:25.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:25.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:25.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:25.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:25.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:25.660+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:26.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:26.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:26.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:26.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:27.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:27.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:27.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:27.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:27.237 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:27.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:27.235110+0000) 2019-10-18T01:44:27.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:44:27.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:27.744375+0000) 2019-10-18T01:44:28.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:28.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:28.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:28.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:29.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:29.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:29.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:29.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:30.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:30.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:30.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:30.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:30.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:30.660+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:31.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:31.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:31.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:31.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:32.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:32.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:32.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:32.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:33.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:33.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:33.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:33.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:33.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:44:33.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:33.168618+0000) 2019-10-18T01:44:34.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:34.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:34.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:34.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:34.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:44:34.120+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:34.122166+0000) 2019-10-18T01:44:35.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:35.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:35.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:35.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:35.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:35.660+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:36.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:36.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:36.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:36.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:37.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:37.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:37.237 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:37.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:37.235294+0000) 2019-10-18T01:44:37.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:44:37.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:37.744562+0000) 2019-10-18T01:44:38.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:38.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:38.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:38.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:39.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:39.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:39.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:39.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:40.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:40.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:40.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:40.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:40.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:40.660+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:41.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:41.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:41.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:41.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:42.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:42.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:42.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:42.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:43.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:43.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:43.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:43.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:43.170 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:44:43.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:43.168711+0000) 2019-10-18T01:44:44.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:44.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:44.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:44.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:44.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:44:44.120+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:44.122267+0000) 2019-10-18T01:44:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:45.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:45.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:45.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:45.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:46.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:46.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:46.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:46.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:47.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:47.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:47.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:47.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:47.238 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:47.233+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:47.235516+0000) 2019-10-18T01:44:47.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:44:47.741+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:47.744732+0000) 2019-10-18T01:44:48.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:48.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:48.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:48.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:49.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:49.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:49.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:49.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:50.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:50.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:50.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:50.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:50.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:50.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:51.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:51.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:51.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:51.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:52.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:52.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:52.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:52.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:53.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:53.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:53.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:53.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:53.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:44:53.164+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:53.168814+0000) 2019-10-18T01:44:54.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:54.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:54.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:54.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:54.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:44:54.120+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:54.122368+0000) 2019-10-18T01:44:55.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:55.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:55.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:55.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:55.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:44:55.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:44:56.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:56.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:56.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:56.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:57.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:57.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:57.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:57.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:57.238 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:57.232+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:57.235692+0000) 2019-10-18T01:44:57.238 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:44:57.232+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:57.236615+0000) 2019-10-18T01:44:57.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:44:57.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:44:57.744908+0000) 2019-10-18T01:44:58.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:58.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:58.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:58.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:44:59.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:59.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:44:59.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:44:59.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:00.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:00.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:00.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:00.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:00.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:00.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:01.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:01.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:01.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:01.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:01.738 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:45:01.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:01.736190+0000) 2019-10-18T01:45:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:02.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:02.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:02.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:03.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:03.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:03.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:03.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:03.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:45:03.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:03.168909+0000) 2019-10-18T01:45:04.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:04.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:04.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:04.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:45:04.119+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:04.122470+0000) 2019-10-18T01:45:05.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:05.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:05.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:05.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:05.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:05.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:06.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:06.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:06.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:06.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:07.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:07.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:07.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:07.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:07.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:45:07.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:07.745089+0000) 2019-10-18T01:45:08.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:08.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:08.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:08.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:09.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:09.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:09.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:09.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:10.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:10.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:10.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:10.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:10.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:10.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:11.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:11.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:11.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:11.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:11.738 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:45:11.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:11.736383+0000) 2019-10-18T01:45:12.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:11.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:12.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:11.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:12.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:12.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:12.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:12.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:13.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:45:13.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:13.169011+0000) 2019-10-18T01:45:13.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:13.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:13.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:13.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:14.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:45:14.119+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:14.122573+0000) 2019-10-18T01:45:15.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:14.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:15.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:14.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:15.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:15.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:15.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:15.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:15.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:15.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:16.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:16.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:16.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:16.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:17.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:45:17.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:17.745241+0000) 2019-10-18T01:45:18.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:18.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:18.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:18.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:19.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:19.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:19.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:19.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:20.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:20.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:20.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:20.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:20.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:20.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:21.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:21.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:21.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:21.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:21.739 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:45:21.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:21.736589+0000) 2019-10-18T01:45:22.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:22.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:22.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:22.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:23.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:23.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:23.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:23.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:23.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:45:23.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:23.169110+0000) 2019-10-18T01:45:24.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:23.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:24.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:23.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:24.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:45:24.119+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:24.122677+0000) 2019-10-18T01:45:24.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:24.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:24.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:24.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:25.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:25.667+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:26.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:25.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:26.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:25.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:26.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:26.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:26.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:26.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:27.747 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:45:27.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:27.745445+0000) 2019-10-18T01:45:28.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:28.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:28.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:28.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:29.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:29.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:29.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:29.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:29.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:29.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:29.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:29.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:30.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:30.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:30.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:30.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:30.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:30.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:31.739 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:45:31.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:31.736756+0000) 2019-10-18T01:45:32.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:32.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:32.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:32.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:33.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:33.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:33.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:33.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:33.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:45:33.163+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:33.169208+0000) 2019-10-18T01:45:34.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:34.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:34.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:34.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:34.124 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:45:34.119+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:34.122777+0000) 2019-10-18T01:45:35.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:35.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:35.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:35.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:35.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:35.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:36.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:36.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:36.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:36.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:37.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:37.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:37.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:37.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:37.748 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:45:37.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:37.745635+0000) 2019-10-18T01:45:38.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:38.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:38.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:38.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:39.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:39.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:39.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:39.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:40.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:40.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:40.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:40.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:40.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:40.670+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:41.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:41.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:41.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:41.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:41.739 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:45:41.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:41.736937+0000) 2019-10-18T01:45:42.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:42.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:42.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:42.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:43.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:43.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:43.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:45:43.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:43.169306+0000) 2019-10-18T01:45:44.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:44.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:44.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:44.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:44.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:45:44.118+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:44.122877+0000) 2019-10-18T01:45:45.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:45.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:45.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:45.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:45.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:45.670+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:46.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:46.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:46.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:46.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:47.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:47.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:47.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:47.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:47.748 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:45:47.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:47.745836+0000) 2019-10-18T01:45:48.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:48.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:48.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:48.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:49.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:49.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:49.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:49.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:50.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:50.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:50.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:50.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:50.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:50.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:51.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:51.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:51.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:51.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:51.739 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:45:51.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:51.737129+0000) 2019-10-18T01:45:52.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:52.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:52.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:52.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:52.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:52.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:52.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:52.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:53.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:45:53.162+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:53.169436+0000) 2019-10-18T01:45:53.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:53.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:53.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:53.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:54.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:45:54.118+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:54.122976+0000) 2019-10-18T01:45:54.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:54.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:54.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:54.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:55.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:45:55.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:45:56.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:55.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:56.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:55.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:56.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:56.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:56.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:56.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:57.748 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:45:57.740+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:45:57.746018+0000) 2019-10-18T01:45:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:58.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:58.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:58.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:45:58.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:58.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:45:58.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:58.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:00.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:59.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:00.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:45:59.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:00.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:00.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:01.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:01.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:01.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:01.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:01.739 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:46:01.732+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:01.737308+0000) 2019-10-18T01:46:02.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:02.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:02.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:02.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:03.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:03.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:03.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:03.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:03.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:46:03.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:03.169540+0000) 2019-10-18T01:46:04.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:04.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:04.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:04.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:04.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:04.118+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:04.123073+0000) 2019-10-18T01:46:05.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:05.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:05.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:05.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:05.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:05.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:05.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:05.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:05.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:05.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:06.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:06.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:06.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:06.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:07.748 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:46:07.744+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:07.746220+0000) 2019-10-18T01:46:07.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:07.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:07.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:07.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:08.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:08.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:08.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:08.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:09.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:09.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:09.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:09.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:10.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:10.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:10.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:10.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:10.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:10.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:11.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:46:11.731+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:11.737465+0000) 2019-10-18T01:46:11.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:11.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:11.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:11.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:12.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:12.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:12.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:12.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:13.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:46:13.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:13.169636+0000) 2019-10-18T01:46:13.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:13.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:13.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:13.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:14.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:14.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:14.123171+0000) 2019-10-18T01:46:14.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:14.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:14.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:14.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:15.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:15.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:15.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:15.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:15.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:15.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:16.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:16.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:16.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:16.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:17.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:46:17.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:17.746397+0000) 2019-10-18T01:46:17.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:17.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:17.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:17.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:18.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:18.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:19.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:19.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:19.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:19.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:20.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:20.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:20.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:20.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:20.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:20.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:21.740 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:46:21.731+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:21.737615+0000) 2019-10-18T01:46:21.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:21.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:21.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:21.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:22.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:22.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:35:48.546671+0000 (oldest deadline 2019-10-18T01:36:08.546671+0000) 2019-10-18T01:46:22.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:22.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:23.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:46:23.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:23.169732+0000) 2019-10-18T01:46:23.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:23.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:24.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:24.117+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:24.123273+0000) 2019-10-18T01:46:24.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:24.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:25.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:25.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:25.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:25.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:26.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:26.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:27.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:46:27.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:27.746582+0000) 2019-10-18T01:46:27.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:27.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:28.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:28.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:29.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:29.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:30.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:30.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:30.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:30.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:31.740 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:46:31.731+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:31.737787+0000) 2019-10-18T01:46:31.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:31.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:32.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:32.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:33.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:46:33.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:33.169825+0000) 2019-10-18T01:46:33.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:33.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:34.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:34.121+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:34.123374+0000) 2019-10-18T01:46:34.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:34.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:35.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:35.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:35.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:35.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:36.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:36.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:37.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:37.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:37.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:46:37.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:37.746761+0000) 2019-10-18T01:46:38.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:38.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:39.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:39.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:40.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:40.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:40.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:40.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:41.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:41.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:41.740 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:46:41.731+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:41.737977+0000) 2019-10-18T01:46:42.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:42.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:43.171 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:46:43.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:43.169923+0000) 2019-10-18T01:46:43.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:43.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:44.125 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:44.121+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:44.123466+0000) 2019-10-18T01:46:44.126 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:44.121+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:44.124866+0000) 2019-10-18T01:46:44.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:44.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:45.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:45.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:45.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:45.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:45.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:45.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:46.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:46.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:46.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:46.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:47.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:46:47.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:47.746952+0000) 2019-10-18T01:46:47.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:47.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:47.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:47.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:48.625 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:48.620+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:48.623653+0000) 2019-10-18T01:46:48.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:48.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:48.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:48.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:49.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:49.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:49.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:49.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:50.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:50.684+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:50.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:50.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:50.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:50.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:51.740 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:46:51.735+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:51.738168+0000) 2019-10-18T01:46:51.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:51.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:51.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:51.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:52.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:52.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:52.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:52.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:53.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:46:53.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:53.170017+0000) 2019-10-18T01:46:53.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:53.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:53.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:53.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:54.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:54.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:54.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:54.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:55.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:46:55.684+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:46:55.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:55.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:55.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:55.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:56.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:56.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:56.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:56.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:57.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:46:57.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:57.747142+0000) 2019-10-18T01:46:57.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:57.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:57.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:57.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:58.625 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:46:58.620+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:46:58.623755+0000) 2019-10-18T01:46:58.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:58.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:58.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:58.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:46:59.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:59.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:46:59.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:46:59.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:00.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:00.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:00.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:00.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:00.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:00.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:01.740 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:47:01.735+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:01.738405+0000) 2019-10-18T01:47:01.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:01.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:01.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:01.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:02.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:02.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:02.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:02.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:03.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:47:03.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:03.170111+0000) 2019-10-18T01:47:03.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:03.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:03.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:03.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:04.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:04.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:04.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:04.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:05.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:05.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:05.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:05.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:05.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:05.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:06.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:06.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:06.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:06.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:07.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:47:07.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:07.747322+0000) 2019-10-18T01:47:07.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:07.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:07.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:07.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:08.625 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:47:08.620+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:08.623859+0000) 2019-10-18T01:47:08.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:08.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:08.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:08.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:09.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:09.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:09.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:09.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:10.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:10.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:10.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:10.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:10.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:10.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:11.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:47:11.735+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:11.738557+0000) 2019-10-18T01:47:11.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:11.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:11.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:11.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:12.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:12.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:12.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:12.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:13.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:47:13.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:13.170210+0000) 2019-10-18T01:47:13.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:13.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:13.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:13.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:14.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:14.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:14.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:14.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:15.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:15.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:15.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:15.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:15.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:15.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:16.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:16.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:16.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:16.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:17.749 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:47:17.743+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:17.747511+0000) 2019-10-18T01:47:17.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:17.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:17.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:17.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:18.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:47:18.620+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:18.623965+0000) 2019-10-18T01:47:18.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:18.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:18.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:18.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:19.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:19.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:19.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:19.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:20.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:20.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:20.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:20.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:20.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:20.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:21.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:47:21.735+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:21.738747+0000) 2019-10-18T01:47:21.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:21.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:21.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:21.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:22.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:22.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:22.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:22.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:23.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:47:23.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:23.170281+0000) 2019-10-18T01:47:23.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:23.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:23.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:23.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:24.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:24.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:24.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:24.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:25.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:25.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:25.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:25.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:25.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:25.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:26.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:26.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:26.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:26.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:27.750 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:47:27.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:27.747693+0000) 2019-10-18T01:47:27.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:27.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:27.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:27.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:28.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:47:28.619+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:28.624066+0000) 2019-10-18T01:47:28.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:28.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:28.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:28.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:29.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:29.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:29.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:29.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:30.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:30.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:30.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:30.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:30.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:30.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:31.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:47:31.734+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:31.738935+0000) 2019-10-18T01:47:31.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:31.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:31.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:31.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:33.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:33.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:33.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:33.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:33.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:47:33.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:33.170375+0000) 2019-10-18T01:47:34.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:34.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:34.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:34.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:35.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:35.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:35.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:35.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:35.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:35.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:36.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:36.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:36.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:36.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:37.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:37.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:37.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:37.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:37.750 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:47:37.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:37.747873+0000) 2019-10-18T01:47:38.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:38.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:38.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:38.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:38.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:47:38.619+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:38.624171+0000) 2019-10-18T01:47:39.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:39.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:39.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:39.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:40.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:40.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:40.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:40.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:40.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:40.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:41.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:41.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:41.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:41.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:41.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:47:41.734+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:41.739136+0000) 2019-10-18T01:47:42.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:42.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:42.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:42.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:43.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:43.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:43.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:43.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:43.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:47:43.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:43.170481+0000) 2019-10-18T01:47:44.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:44.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:44.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:44.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:45.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:45.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:45.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:45.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:45.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:45.691+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:46.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:46.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:46.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:46.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:47.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:47.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:47.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:47.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:47.751 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:47:47.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:47.748056+0000) 2019-10-18T01:47:48.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:48.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:48.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:48.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:48.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:47:48.619+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:48.624278+0000) 2019-10-18T01:47:49.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:49.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:49.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:49.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:50.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:50.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:50.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:50.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:50.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:50.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:51.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:51.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:51.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:51.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:51.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:47:51.734+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:51.739351+0000) 2019-10-18T01:47:52.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:52.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:52.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:52.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:53.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:53.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:53.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:53.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:53.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:47:53.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:53.170581+0000) 2019-10-18T01:47:53.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:53.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:53.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:53.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:55.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:55.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:55.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:55.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:55.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:47:55.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:47:56.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:56.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:56.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:56.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:57.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:56.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:57.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:56.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:57.751 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:47:57.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:57.748236+0000) 2019-10-18T01:47:57.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:57.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:57.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:57.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:58.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:47:58.619+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:47:58.624382+0000) 2019-10-18T01:47:58.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:58.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:58.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:58.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:47:59.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:59.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:47:59.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:47:59.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:00.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:00.694+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:01.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:00.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:01.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:00.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:01.741 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:48:01.734+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:01.739540+0000) 2019-10-18T01:48:01.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:01.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:01.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:01.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:02.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:02.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:02.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:02.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:03.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:48:03.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:03.170676+0000) 2019-10-18T01:48:04.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:03.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:04.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:03.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:05.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:05.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:05.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:05.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:05.703 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:05.694+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:05.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:05.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:05.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:05.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:06.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:06.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:06.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:06.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:07.750 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:48:07.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:07.748448+0000) 2019-10-18T01:48:07.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:07.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:07.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:07.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:08.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:48:08.618+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:08.624485+0000) 2019-10-18T01:48:08.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:08.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:08.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:08.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:09.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:09.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:09.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:09.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:10.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:10.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:10.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:10.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:10.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:10.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:11.742 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:48:11.734+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:11.739758+0000) 2019-10-18T01:48:11.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:11.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:11.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:11.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:12.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:12.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:12.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:12.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:13.172 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:48:13.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:13.170789+0000) 2019-10-18T01:48:13.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:13.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:13.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:13.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:14.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:14.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:14.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:14.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:15.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:15.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:15.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:15.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:15.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:15.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:16.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:16.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:16.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:16.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:17.751 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:48:17.742+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:17.748633+0000) 2019-10-18T01:48:17.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:17.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:17.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:17.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:18.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:48:18.618+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:18.624589+0000) 2019-10-18T01:48:18.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:18.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:18.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:18.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:19.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:19.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:19.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:19.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:20.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:20.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:20.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:20.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:20.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:20.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:21.742 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:48:21.734+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:21.739975+0000) 2019-10-18T01:48:21.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:21.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:21.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:21.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:22.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:22.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:22.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:22.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:23.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:48:23.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:23.170884+0000) 2019-10-18T01:48:23.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:23.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:23.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:23.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:24.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:24.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:24.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:24.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:25.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:25.698+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:25.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:25.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:25.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:25.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:26.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:26.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:26.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:26.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:27.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:48:27.746+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:27.748846+0000) 2019-10-18T01:48:27.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:27.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:27.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:27.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:28.626 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:48:28.618+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:28.624694+0000) 2019-10-18T01:48:28.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:28.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:28.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:28.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:29.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:29.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:29.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:29.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:30.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:30.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:30.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:30.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:30.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:30.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:31.742 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:48:31.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:31.740190+0000) 2019-10-18T01:48:31.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:31.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:31.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:31.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:32.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:32.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:32.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:32.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:33.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:48:33.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:33.170977+0000) 2019-10-18T01:48:33.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:33.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:33.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:33.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:34.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:34.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:34.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:34.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:35.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:35.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:35.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:35.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:35.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:35.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:36.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:36.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:36.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:36.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:37.751 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:48:37.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:37.749030+0000) 2019-10-18T01:48:37.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:37.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:37.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:37.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:38.627 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:48:38.617+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:38.624803+0000) 2019-10-18T01:48:38.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:38.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:38.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:38.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:39.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:39.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:39.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:39.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:40.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:40.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:40.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:40.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:40.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:40.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:41.742 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:48:41.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:41.740387+0000) 2019-10-18T01:48:41.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:41.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:41.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:41.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:42.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:42.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:42.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:42.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:43.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:48:43.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:43.171071+0000) 2019-10-18T01:48:43.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:43.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:43.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:43.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:44.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:44.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:44.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:44.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:45.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:45.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:45.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:45.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:45.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:45.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:46.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:46.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:46.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:46.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:47.751 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:48:47.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:47.749218+0000) 2019-10-18T01:48:47.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:47.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:47.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:47.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:48.627 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:48:48.621+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:48.625464+0000) 2019-10-18T01:48:49.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:49.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:49.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:49.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:50.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:50.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:50.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:50.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:50.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:50.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:51.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:51.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:51.743 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:48:51.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:51.740579+0000) 2019-10-18T01:48:52.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:52.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:52.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:52.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:53.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:53.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:53.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:53.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:53.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:48:53.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:53.171173+0000) 2019-10-18T01:48:54.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:54.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:54.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:54.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:55.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:55.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:55.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:55.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:55.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:48:55.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:48:56.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:56.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:56.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:56.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:57.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:57.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:57.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:57.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:57.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:48:57.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:57.749359+0000) 2019-10-18T01:48:58.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:58.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:58.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:58.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:48:58.627 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:48:58.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:48:58.625565+0000) 2019-10-18T01:48:59.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:59.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:48:59.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:48:59.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:00.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:00.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:00.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:00.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:00.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:00.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:01.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:01.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:01.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:01.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:01.743 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:49:01.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:01.740798+0000) 2019-10-18T01:49:02.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:02.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:02.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:02.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:03.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:03.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:03.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:03.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:03.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:49:03.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:03.171268+0000) 2019-10-18T01:49:04.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:04.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:04.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:04.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:05.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:05.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:05.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:05.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:05.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:05.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:06.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:06.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:06.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:06.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:07.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:07.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:07.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:07.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:07.751 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:49:07.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:07.749542+0000) 2019-10-18T01:49:08.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:08.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:08.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:08.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:08.627 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:49:08.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:08.625668+0000) 2019-10-18T01:49:09.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:09.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:09.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:09.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:10.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:10.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:10.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:10.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:11.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:11.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:11.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:11.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:11.745 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:49:11.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:11.741019+0000) 2019-10-18T01:49:12.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:12.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:12.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:12.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:13.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:49:13.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:13.171365+0000) 2019-10-18T01:49:13.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:13.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:13.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:13.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:14.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:14.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:14.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:14.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:15.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:15.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:15.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:15.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:15.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:15.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:16.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:16.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:16.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:16.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:17.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:17.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:17.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:17.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:17.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:49:17.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:17.749693+0000) 2019-10-18T01:49:18.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:18.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:18.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:18.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:18.627 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:49:18.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:18.625771+0000) 2019-10-18T01:49:19.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:19.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:19.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:19.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:20.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:20.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:20.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:20.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:20.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:20.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:21.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:21.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:21.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:21.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:21.743 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:49:21.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:21.741221+0000) 2019-10-18T01:49:22.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:22.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:22.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:22.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:23.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:49:23.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:23.171462+0000) 2019-10-18T01:49:23.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:23.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:23.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:23.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:24.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:24.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:24.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:24.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:25.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:25.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:25.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:25.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:25.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:25.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:26.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:26.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:26.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:26.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:27.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:27.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:27.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:27.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:27.752 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:49:27.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:27.749876+0000) 2019-10-18T01:49:28.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:28.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:28.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:28.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:28.627 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:49:28.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:28.625868+0000) 2019-10-18T01:49:29.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:29.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:29.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:29.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:30.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:30.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:30.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:30.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:30.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:30.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:31.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:31.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:31.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:31.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:31.743 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:49:31.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:31.741438+0000) 2019-10-18T01:49:32.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:32.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:32.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:32.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:33.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:49:33.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:33.171562+0000) 2019-10-18T01:49:33.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:33.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:33.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:33.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:34.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:34.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:34.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:34.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:35.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:35.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:35.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:35.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:35.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:35.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:36.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:36.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:36.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:36.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:37.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:37.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:37.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:37.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:37.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:49:37.745+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:37.750057+0000) 2019-10-18T01:49:38.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:38.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:38.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:38.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:38.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:49:38.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:38.625974+0000) 2019-10-18T01:49:39.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:39.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:39.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:39.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:40.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:40.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:40.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:40.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:40.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:40.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:41.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:41.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:41.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:41.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:41.744 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:49:41.737+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:41.741663+0000) 2019-10-18T01:49:42.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:42.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:42.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:42.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:43.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:49:43.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:43.171659+0000) 2019-10-18T01:49:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:43.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:43.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:43.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:44.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:44.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:44.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:44.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:45.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:45.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:45.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:45.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:45.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:45.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:46.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:46.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:46.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:46.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:47.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:47.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:47.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:47.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:47.752 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:49:47.749+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:47.750254+0000) 2019-10-18T01:49:48.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:48.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:48.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:48.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:48.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:49:48.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:48.626078+0000) 2019-10-18T01:49:49.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:49.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:49.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:49.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:50.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:50.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:50.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:50.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:50.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:50.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:51.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:51.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:51.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:51.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:51.744 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:49:51.736+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:51.741871+0000) 2019-10-18T01:49:52.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:52.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:52.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:52.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:53.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:49:53.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:53.171758+0000) 2019-10-18T01:49:53.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:53.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:53.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:53.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:54.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:54.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:54.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:54.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:55.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:55.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:55.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:55.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:55.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:49:55.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:49:56.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:56.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:56.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:56.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:57.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:57.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:57.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:57.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:57.752 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:49:57.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:57.750443+0000) 2019-10-18T01:49:58.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:49:58.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:49:58.626181+0000) 2019-10-18T01:49:58.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:58.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:58.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:58.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:49:59.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:59.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:49:59.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:49:59.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:00.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:00.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:00.723 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:00.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:01.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:01.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:01.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:01.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:01.744 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:50:01.736+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:01.742086+0000) 2019-10-18T01:50:02.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:02.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:02.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:02.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:03.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:50:03.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:03.171855+0000) 2019-10-18T01:50:03.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:03.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:03.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:03.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:04.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:04.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:04.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:04.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:05.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:05.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:05.724 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:05.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:06.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:06.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:06.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:06.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:07.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:07.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:07.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:07.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:07.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:50:07.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:07.750632+0000) 2019-10-18T01:50:08.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:08.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:08.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:08.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:08.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:50:08.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:08.626285+0000) 2019-10-18T01:50:09.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:09.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:09.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:09.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:10.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:10.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:10.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:10.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:10.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:10.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:11.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:11.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:11.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:11.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:11.744 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:50:11.740+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:11.742324+0000) 2019-10-18T01:50:12.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:12.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:12.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:12.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:13.173 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:50:13.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:13.171954+0000) 2019-10-18T01:50:13.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:13.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:13.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:13.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:14.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:14.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:14.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:14.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:15.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:15.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:15.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:15.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:15.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:15.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:16.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:16.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:16.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:16.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:17.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:17.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:17.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:17.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:17.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:50:17.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:17.750845+0000) 2019-10-18T01:50:18.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:50:18.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:18.626385+0000) 2019-10-18T01:50:18.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:18.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:18.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:18.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:19.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:19.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:19.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:19.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:20.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:20.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:20.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:20.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:20.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:20.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:21.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:21.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:21.744 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:50:21.740+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:21.742539+0000) 2019-10-18T01:50:22.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:22.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:22.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:22.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:23.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:50:23.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:23.172052+0000) 2019-10-18T01:50:23.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:23.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:23.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:23.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:24.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:24.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:25.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:25.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:25.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:25.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:25.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:25.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:26.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:26.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:26.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:26.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:27.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:27.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:27.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:27.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:27.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:50:27.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:27.751049+0000) 2019-10-18T01:50:28.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:28.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:28.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:28.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:28.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:50:28.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:28.626484+0000) 2019-10-18T01:50:29.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:29.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:29.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:29.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:30.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:30.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:30.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:30.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:30.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:30.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:31.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:31.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:31.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:31.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:31.745 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:50:31.740+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:31.742753+0000) 2019-10-18T01:50:32.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:32.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:32.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:32.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:33.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:50:33.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:33.172151+0000) 2019-10-18T01:50:33.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:33.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:33.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:33.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:34.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:34.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:34.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:34.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:35.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:35.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:35.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:35.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:35.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:35.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:36.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:36.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:36.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:36.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:37.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:37.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:37.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:37.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:37.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:50:37.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:37.751235+0000) 2019-10-18T01:50:38.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:38.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:38.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:38.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:38.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:50:38.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:38.626590+0000) 2019-10-18T01:50:39.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:39.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:39.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:39.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:40.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:40.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:40.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:40.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:40.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:40.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:41.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:41.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:41.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:41.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:41.745 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:50:41.740+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:41.742952+0000) 2019-10-18T01:50:42.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:42.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:42.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:42.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:43.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:50:43.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:43.172249+0000) 2019-10-18T01:50:43.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:43.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:43.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:43.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:44.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:44.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:44.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:44.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:45.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:45.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:45.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:45.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:45.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:45.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:46.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:46.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:46.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:46.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:47.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:47.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:47.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:47.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:47.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:50:47.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:47.751427+0000) 2019-10-18T01:50:48.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:48.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:48.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:48.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:48.628 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:50:48.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:48.626691+0000) 2019-10-18T01:50:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:49.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:49.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:50.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:50.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:50.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:50.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:50.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:50.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:51.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:51.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:51.745 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:50:51.740+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:51.743165+0000) 2019-10-18T01:50:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:52.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:52.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:53.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:50:53.166+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:53.172350+0000) 2019-10-18T01:50:53.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:53.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:53.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:53.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:54.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:54.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:54.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:54.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:55.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:55.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:55.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:55.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:55.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:50:55.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:50:56.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:56.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:56.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:56.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:57.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:57.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:57.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:57.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:57.753 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:50:57.748+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:57.751612+0000) 2019-10-18T01:50:58.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:58.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:58.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:58.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:50:58.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:50:58.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:50:58.626796+0000) 2019-10-18T01:50:59.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:59.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:50:59.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:50:59.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:00.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:00.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:00.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:00.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:00.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:00.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:01.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:01.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:01.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:01.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:01.745 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:51:01.740+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:01.743380+0000) 2019-10-18T01:51:02.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:02.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:02.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:02.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:03.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:51:03.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:03.172448+0000) 2019-10-18T01:51:03.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:03.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:03.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:03.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:04.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:04.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:04.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:04.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:05.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:05.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:05.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:05.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:05.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:05.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:06.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:06.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:06.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:06.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:07.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:07.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:07.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:07.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:07.754 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:51:07.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:07.751793+0000) 2019-10-18T01:51:08.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:08.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:08.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:08.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:08.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:51:08.621+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:08.626901+0000) 2019-10-18T01:51:09.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:09.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:09.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:09.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:10.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:10.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:10.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:10.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:10.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:10.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:11.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:11.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:11.745 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:51:11.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:11.743601+0000) 2019-10-18T01:51:12.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:12.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:12.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:12.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:13.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:51:13.165+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:13.172546+0000) 2019-10-18T01:51:13.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:13.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:13.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:13.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:14.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:14.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:14.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:15.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:15.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:15.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:15.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:15.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:15.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:16.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:16.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:16.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:16.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:17.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:17.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:17.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:17.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:17.755 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:51:17.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:17.751973+0000) 2019-10-18T01:51:18.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:18.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:18.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:18.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:18.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:51:18.621+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:18.627014+0000) 2019-10-18T01:51:19.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:19.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:19.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:19.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:20.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:20.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:20.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:20.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:20.737 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:20.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:21.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:21.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:21.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:21.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:21.746 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:51:21.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:21.743815+0000) 2019-10-18T01:51:22.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:22.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:22.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:22.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:23.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:51:23.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:23.172643+0000) 2019-10-18T01:51:23.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:23.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:23.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:23.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:24.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:24.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:24.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:24.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:25.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:25.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:25.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:25.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:25.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:25.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:26.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:26.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:26.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:26.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:27.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:27.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:27.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:27.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:27.754 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:51:27.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:27.752157+0000) 2019-10-18T01:51:28.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:28.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:28.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:28.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:28.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:51:28.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:28.627119+0000) 2019-10-18T01:51:29.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:29.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:29.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:29.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:30.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:30.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:30.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:30.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:30.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:30.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:31.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:31.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:31.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:31.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:31.746 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:51:31.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:31.744039+0000) 2019-10-18T01:51:32.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:32.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:32.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:32.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:33.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:33.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:33.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:33.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:33.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:51:33.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:33.172743+0000) 2019-10-18T01:51:34.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:34.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:34.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:34.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:35.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:35.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:35.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:35.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:35.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:35.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:36.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:36.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:36.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:36.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:37.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:37.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:37.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:37.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:37.754 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:51:37.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:37.752340+0000) 2019-10-18T01:51:38.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:38.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:38.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:38.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:38.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:51:38.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:38.627226+0000) 2019-10-18T01:51:39.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:39.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:39.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:39.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:40.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:40.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:40.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:40.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:40.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:40.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:41.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:41.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:41.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:41.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:41.746 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:51:41.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:41.744224+0000) 2019-10-18T01:51:42.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:42.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:42.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:42.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:43.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:43.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:43.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:43.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:43.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:51:43.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:43.172840+0000) 2019-10-18T01:51:44.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:44.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:44.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:44.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:45.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:45.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:45.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:45.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:45.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:45.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:45.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:45.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:47.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:47.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:47.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:47.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:47.754 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:51:47.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:47.752550+0000) 2019-10-18T01:51:48.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:48.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:48.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:48.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:48.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:51:48.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:48.627327+0000) 2019-10-18T01:51:49.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:49.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:49.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:49.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:50.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:50.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:50.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:50.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:50.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:50.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:50.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:50.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:50.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:50.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:51.746 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:51:51.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:51.744410+0000) 2019-10-18T01:51:51.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:51.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:51.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:51.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:53.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:53.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:53.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:53.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:53.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:51:53.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:53.172941+0000) 2019-10-18T01:51:53.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:53.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:53.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:53.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:54.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:54.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:54.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:54.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:55.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:51:55.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:51:55.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:55.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:55.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:55.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:56.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:56.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:56.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:56.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:57.755 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:51:57.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:57.752732+0000) 2019-10-18T01:51:57.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:57.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:57.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:57.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:58.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:51:58.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:51:58.627433+0000) 2019-10-18T01:51:58.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:58.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:58.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:58.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:51:59.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:59.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:51:59.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:51:59.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:00.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:00.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:00.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:00.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:00.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:00.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:01.747 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:52:01.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:01.744624+0000) 2019-10-18T01:52:01.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:01.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:01.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:01.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:02.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:02.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:02.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:02.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:03.174 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:52:03.168+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:03.173037+0000) 2019-10-18T01:52:03.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:03.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:03.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:03.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:04.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:04.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:04.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:04.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:05.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:05.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:05.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:05.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:05.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:05.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:06.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:06.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:06.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:06.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:07.755 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:52:07.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:07.752919+0000) 2019-10-18T01:52:08.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:08.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:08.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:08.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:08.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:52:08.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:08.627543+0000) 2019-10-18T01:52:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:09.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:09.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:10.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:10.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:10.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:10.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:10.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:10.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:11.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:11.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:11.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:11.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:11.747 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:52:11.739+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:11.744809+0000) 2019-10-18T01:52:12.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:12.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:12.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:12.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:13.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:13.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:13.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:13.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:13.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:52:13.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:13.173138+0000) 2019-10-18T01:52:14.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:14.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:14.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:14.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:15.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:15.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:15.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:15.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:15.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:15.739+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:16.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:16.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:16.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:16.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:17.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:17.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:17.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:17.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:17.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:52:17.747+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:17.753104+0000) 2019-10-18T01:52:18.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:18.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:18.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:18.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:18.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:52:18.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:18.627651+0000) 2019-10-18T01:52:19.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:19.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:19.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:19.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:20.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:20.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:20.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:20.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:20.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:20.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:21.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:21.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:21.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:21.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:21.747 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:52:21.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:21.745023+0000) 2019-10-18T01:52:22.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:22.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:22.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:22.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:23.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:23.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:23.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:23.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:23.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:52:23.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:23.173237+0000) 2019-10-18T01:52:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:24.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:24.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:25.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:25.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:25.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:25.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:25.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:25.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:26.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:26.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:27.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:27.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:27.755 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:52:27.746+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:27.753298+0000) 2019-10-18T01:52:28.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:28.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:28.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:28.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:28.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:52:28.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:28.627734+0000) 2019-10-18T01:52:29.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:29.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:29.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:29.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:30.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:30.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:30.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:30.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:30.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:30.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:31.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:31.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:31.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:31.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:31.747 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:52:31.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:31.745206+0000) 2019-10-18T01:52:32.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:32.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:32.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:32.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:33.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:33.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:33.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:33.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:33.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:52:33.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:33.173332+0000) 2019-10-18T01:52:34.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:34.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:34.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:34.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:35.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:35.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:35.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:35.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:35.750 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:35.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:36.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:36.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:36.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:36.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:37.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:37.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:37.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:37.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:37.755 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:52:37.746+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:37.753482+0000) 2019-10-18T01:52:38.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:38.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:38.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:38.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:38.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:52:38.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:38.627841+0000) 2019-10-18T01:52:38.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:38.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:38.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:38.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:39.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:39.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:39.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:39.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:40.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:40.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:40.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:40.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:40.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:40.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:41.747 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:52:41.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:41.745429+0000) 2019-10-18T01:52:42.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:42.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:42.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:42.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:43.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:42.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:43.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:42.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:43.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:52:43.167+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:43.173430+0000) 2019-10-18T01:52:44.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:44.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:44.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:44.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:45.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:44.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:45.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:44.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:45.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:45.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:46.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:46.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:46.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:46.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:47.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:47.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:47.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:47.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:47.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:52:47.746+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:47.753695+0000) 2019-10-18T01:52:48.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:48.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:48.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:48.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:48.629 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:52:48.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:48.627944+0000) 2019-10-18T01:52:49.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:49.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:49.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:49.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:49.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:49.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:49.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:49.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:50.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:50.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:50.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:50.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:50.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:50.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:51.753 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:52:51.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:51.745611+0000) 2019-10-18T01:52:51.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:51.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:51.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:51.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:52.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:52.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:52.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:52.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:53.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:52:53.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:53.173538+0000) 2019-10-18T01:52:53.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:53.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:53.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:53.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:54.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:54.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:54.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:54.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:55.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:52:55.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:52:55.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:55.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:55.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:55.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:56.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:56.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:56.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:56.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:57.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:52:57.746+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:57.753877+0000) 2019-10-18T01:52:57.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:57.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:57.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:57.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:58.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:52:58.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:52:58.628040+0000) 2019-10-18T01:52:58.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:58.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:58.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:58.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:52:59.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:59.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:52:59.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:52:59.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:00.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:00.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:00.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:00.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:00.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:00.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:01.748 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:53:01.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:01.745825+0000) 2019-10-18T01:53:01.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:01.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:01.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:01.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:02.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:02.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:02.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:02.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:03.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:53:03.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:03.173627+0000) 2019-10-18T01:53:03.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:03.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:03.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:03.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:04.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:04.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:04.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:04.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:05.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:05.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:05.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:05.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:05.755 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:05.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:06.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:06.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:06.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:06.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:07.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:53:07.746+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:07.754089+0000) 2019-10-18T01:53:07.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:07.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:07.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:07.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:08.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:53:08.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:08.628144+0000) 2019-10-18T01:53:08.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:08.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:08.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:08.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:09.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:09.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:09.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:09.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:10.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:10.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:10.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:10.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:10.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:10.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:11.748 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:53:11.738+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:11.746039+0000) 2019-10-18T01:53:11.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:11.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:11.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:11.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:12.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:12.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:12.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:12.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:13.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:53:13.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:13.173721+0000) 2019-10-18T01:53:13.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:13.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:13.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:13.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:14.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:14.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:14.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:14.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:15.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:15.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:15.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:15.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:15.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:15.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:16.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:16.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:16.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:16.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:17.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:53:17.750+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:17.754267+0000) 2019-10-18T01:53:17.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:17.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:17.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:17.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:18.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:53:18.622+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:18.628249+0000) 2019-10-18T01:53:18.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:18.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:18.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:18.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:19.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:19.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:20.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:20.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:20.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:20.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:20.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:20.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:21.748 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:53:21.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:21.746262+0000) 2019-10-18T01:53:21.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:21.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:21.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:21.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:22.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:22.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:22.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:22.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:23.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:53:23.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:23.173818+0000) 2019-10-18T01:53:24.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:24.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:24.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:24.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:24.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:24.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:24.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:24.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:25.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:25.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:26.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:26.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:26.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:26.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:27.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:27.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:27.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:27.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:27.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:53:27.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:27.754452+0000) 2019-10-18T01:53:27.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:27.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:27.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:27.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:28.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:53:28.621+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:28.628366+0000) 2019-10-18T01:53:28.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:28.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:28.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:28.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:29.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:29.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:29.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:29.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:30.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:30.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:31.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:31.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:31.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:31.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:31.748 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:53:31.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:31.746472+0000) 2019-10-18T01:53:31.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:31.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:31.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:31.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:32.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:32.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:32.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:32.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:33.175 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:53:33.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:33.173913+0000) 2019-10-18T01:53:34.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:34.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:34.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:34.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:34.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:34.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:34.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:34.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:35.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:35.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:36.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:36.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:36.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:36.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:37.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:37.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:37.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:37.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:37.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:53:37.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:37.754637+0000) 2019-10-18T01:53:38.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:38.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:38.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:38.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:38.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:53:38.621+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:38.628475+0000) 2019-10-18T01:53:39.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:39.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:39.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:39.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:40.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:40.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:40.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:40.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:40.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:40.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:41.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:41.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:41.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:41.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:41.749 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:53:41.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:41.746661+0000) 2019-10-18T01:53:42.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:42.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:42.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:42.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:43.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:43.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:43.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:43.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:43.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:53:43.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:43.174009+0000) 2019-10-18T01:53:44.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:44.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:44.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:44.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:45.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:45.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:45.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:45.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:45.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:45.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:46.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:46.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:46.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:46.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:47.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:47.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:47.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:47.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:47.756 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:53:47.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:47.754817+0000) 2019-10-18T01:53:48.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:48.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:48.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:48.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:48.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:53:48.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:48.628583+0000) 2019-10-18T01:53:49.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:49.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:49.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:49.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:50.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:50.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:50.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:50.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:50.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:50.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:51.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:50.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:51.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:50.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:51.749 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:53:51.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:51.746850+0000) 2019-10-18T01:53:52.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:52.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:52.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:52.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:53.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:53.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:53.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:53.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:53.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:53:53.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:53.174107+0000) 2019-10-18T01:53:54.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:54.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:54.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:54.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:55.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:55.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:55.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:55.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:55.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:53:55.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:53:56.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:56.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:56.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:56.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:57.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:56.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:57.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:56.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:57.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:53:57.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:57.755013+0000) 2019-10-18T01:53:57.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:57.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:57.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:57.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:53:58.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:53:58.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:53:58.628686+0000) 2019-10-18T01:53:58.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:58.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:53:58.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:53:58.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:00.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:00.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:00.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:00.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:00.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:00.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:01.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:01.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:01.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:01.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:01.749 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:54:01.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:01.747064+0000) 2019-10-18T01:54:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:02.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:02.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:03.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:03.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:03.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:03.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:03.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:54:03.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:03.174202+0000) 2019-10-18T01:54:04.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:04.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:04.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:04.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:04.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:04.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:04.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:04.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:05.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:05.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:05.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:05.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:05.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:05.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:07.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:07.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:07.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:07.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:07.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:54:07.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:07.755205+0000) 2019-10-18T01:54:08.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:08.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:08.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:08.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:08.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:54:08.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:08.628794+0000) 2019-10-18T01:54:09.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:09.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:09.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:09.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:10.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:10.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:10.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:10.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:10.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:10.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:11.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:11.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:11.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:11.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:11.749 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:54:11.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:11.747249+0000) 2019-10-18T01:54:11.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:11.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:11.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:11.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:13.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:13.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:13.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:13.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:13.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:54:13.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:13.174303+0000) 2019-10-18T01:54:14.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:14.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:14.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:14.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:15.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:15.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:15.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:15.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:15.767 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:15.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:16.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:16.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:16.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:16.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:17.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:17.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:17.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:17.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:17.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:54:17.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:17.755386+0000) 2019-10-18T01:54:18.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:18.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:18.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:18.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:18.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:54:18.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:18.628898+0000) 2019-10-18T01:54:19.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:19.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:19.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:19.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:20.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:20.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:20.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:20.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:20.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:20.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:21.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:21.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:21.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:21.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:21.749 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:54:21.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:21.747434+0000) 2019-10-18T01:54:22.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:22.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:22.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:22.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:23.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:54:23.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:23.174398+0000) 2019-10-18T01:54:23.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:23.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:23.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:23.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:24.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:24.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:24.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:24.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:25.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:25.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:25.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:25.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:25.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:25.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:26.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:26.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:26.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:26.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:27.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:27.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:27.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:27.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:27.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:54:27.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:27.755577+0000) 2019-10-18T01:54:28.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:28.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:28.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:28.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:28.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:54:28.624+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:28.628998+0000) 2019-10-18T01:54:29.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:29.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:29.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:29.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:30.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:30.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:30.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:30.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:30.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:30.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:31.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:31.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:31.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:31.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:31.749 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:54:31.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:31.747620+0000) 2019-10-18T01:54:32.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:32.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:32.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:32.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:33.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:33.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:33.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:33.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:33.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:54:33.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:33.174493+0000) 2019-10-18T01:54:34.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:34.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:34.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:34.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:35.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:35.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:35.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:35.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:35.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:35.763+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:36.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:36.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:36.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:36.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:37.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:37.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:37.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:37.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:37.757 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:54:37.753+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:37.755762+0000) 2019-10-18T01:54:38.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:38.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:38.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:38.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:38.630 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:54:38.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:38.629101+0000) 2019-10-18T01:54:39.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:39.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:39.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:39.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:40.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:40.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:40.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:40.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:40.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:40.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:41.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:41.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:41.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:41.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:41.750 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:54:41.745+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:41.747835+0000) 2019-10-18T01:54:42.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:42.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:42.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:42.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:43.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:43.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:43.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:43.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:43.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:54:43.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:43.174590+0000) 2019-10-18T01:54:44.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:44.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:44.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:44.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:45.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:45.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:45.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:45.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:45.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:45.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:46.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:46.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:46.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:46.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:47.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:47.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:47.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:47.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:47.758 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:54:47.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:47.755945+0000) 2019-10-18T01:54:48.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:48.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:48.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:48.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:48.635 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:54:48.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:48.629203+0000) 2019-10-18T01:54:49.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:49.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:49.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:49.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:50.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:50.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:50.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:50.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:51.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:51.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:51.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:51.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:51.750 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:54:51.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:51.748020+0000) 2019-10-18T01:54:52.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:52.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:52.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:52.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:53.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:53.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:53.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:53.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:53.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:54:53.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:53.174680+0000) 2019-10-18T01:54:54.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:54.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:54.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:54.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:55.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:55.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:55.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:55.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:55.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:54:55.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:54:56.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:56.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:56.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:56.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:57.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:57.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:57.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:57.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:57.758 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:54:57.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:57.756155+0000) 2019-10-18T01:54:58.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:58.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:58.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:58.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:54:58.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:54:58.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:54:58.629299+0000) 2019-10-18T01:54:59.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:59.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:54:59.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:54:59.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:00.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:00.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:00.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:00.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:00.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:00.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:01.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:00.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:01.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:00.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:01.750 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:55:01.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:01.748240+0000) 2019-10-18T01:55:02.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:02.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:02.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:02.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:02.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:02.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:02.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:02.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:03.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:55:03.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:03.174773+0000) 2019-10-18T01:55:03.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:03.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:03.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:03.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:05.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:05.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:05.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:05.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:05.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:05.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:06.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:06.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:06.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:06.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:07.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:07.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:07.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:07.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:07.758 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:55:07.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:07.756364+0000) 2019-10-18T01:55:08.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:08.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:08.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:08.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:08.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:55:08.623+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:08.629397+0000) 2019-10-18T01:55:08.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:08.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:08.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:08.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:10.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:10.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:10.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:10.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:10.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:10.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:10.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:10.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:10.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:10.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:11.750 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:55:11.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:11.748459+0000) 2019-10-18T01:55:12.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:11.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:12.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:11.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:13.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:13.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:13.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:13.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:13.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:55:13.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:13.174869+0000) 2019-10-18T01:55:13.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:13.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:13.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:13.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:15.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:15.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:15.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:15.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:15.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:15.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:16.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:16.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:16.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:16.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:17.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:17.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:17.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:17.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:17.759 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:55:17.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:17.756547+0000) 2019-10-18T01:55:18.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:18.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:18.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:18.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:18.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:55:18.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:18.629472+0000) 2019-10-18T01:55:19.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:18.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:19.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:18.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:19.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:19.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:19.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:19.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:20.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:20.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:20.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:20.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:20.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:20.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:21.752 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:55:21.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:21.748677+0000) 2019-10-18T01:55:21.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:21.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:21.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:21.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:22.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:22.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:22.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:22.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:23.176 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:55:23.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:23.174967+0000) 2019-10-18T01:55:23.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:23.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:23.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:23.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:25.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:24.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:25.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:24.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:25.779 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:25.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:25.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:25.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:25.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:25.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:26.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:26.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:26.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:26.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:27.758 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:55:27.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:27.756729+0000) 2019-10-18T01:55:28.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:28.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:28.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:28.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:28.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:55:28.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:28.629570+0000) 2019-10-18T01:55:28.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:28.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:28.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:28.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:29.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:29.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:29.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:29.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:30.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:30.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:30.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:30.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:30.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:30.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:31.751 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:55:31.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:31.748892+0000) 2019-10-18T01:55:31.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:31.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:31.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:31.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:33.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:33.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:33.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:33.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:33.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:55:33.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:33.175067+0000) 2019-10-18T01:55:33.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:33.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:33.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:33.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:34.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:34.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:34.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:34.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:35.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:35.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:35.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:35.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:35.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:35.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:36.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:36.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:36.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:36.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:37.759 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:55:37.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:37.756918+0000) 2019-10-18T01:55:37.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:37.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:37.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:37.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:38.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:55:38.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:38.629672+0000) 2019-10-18T01:55:38.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:38.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:38.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:38.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:39.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:39.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:39.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:39.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:40.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:40.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:40.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:40.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:40.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:40.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:41.751 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:55:41.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:41.749107+0000) 2019-10-18T01:55:41.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:41.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:41.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:41.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:42.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:42.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:42.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:42.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:43.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:55:43.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:43.175166+0000) 2019-10-18T01:55:43.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:43.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:43.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:43.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:44.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:44.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:44.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:44.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:45.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:45.773+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:45.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:45.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:45.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:45.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:46.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:46.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:46.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:46.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:47.759 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:55:47.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:47.757100+0000) 2019-10-18T01:55:47.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:47.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:47.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:47.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:48.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:55:48.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:48.629779+0000) 2019-10-18T01:55:48.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:48.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:48.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:48.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:49.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:49.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:49.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:49.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:50.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:50.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:50.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:50.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:50.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:50.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:51.751 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:55:51.744+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:51.749305+0000) 2019-10-18T01:55:51.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:51.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:51.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:51.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:52.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:52.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:52.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:53.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:55:53.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:53.175265+0000) 2019-10-18T01:55:53.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:53.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:53.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:53.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:54.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:54.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:54.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:54.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:55.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:55:55.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:55:55.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:55.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:55.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:55.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:56.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:56.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:56.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:56.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:57.759 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:55:57.752+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:57.757282+0000) 2019-10-18T01:55:57.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:57.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:57.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:57.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:55:58.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:55:58.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:55:58.629880+0000) 2019-10-18T01:55:58.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:58.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:55:58.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:55:58.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:00.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:00.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:00.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:00.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:00.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:00.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:00.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:00.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:00.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:00.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:01.751 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:56:01.743+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:01.749508+0000) 2019-10-18T01:56:01.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:01.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:01.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:01.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:02.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:02.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:02.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:02.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:03.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:56:03.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:03.175367+0000) 2019-10-18T01:56:03.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:03.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:03.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:03.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:05.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:05.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:05.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:05.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:05.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:05.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:06.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:06.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:06.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:06.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:07.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:07.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:07.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:07.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:07.759 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:56:07.751+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:07.757492+0000) 2019-10-18T01:56:08.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:08.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:08.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:08.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:08.631 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:56:08.629+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:08.629985+0000) 2019-10-18T01:56:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:09.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:09.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:10.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:10.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:10.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:10.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:10.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:10.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:11.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:11.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:11.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:11.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:11.751 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:56:11.743+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:11.749706+0000) 2019-10-18T01:56:12.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:12.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:12.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:12.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:13.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:56:13.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:13.175467+0000) 2019-10-18T01:56:13.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:13.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:13.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:13.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:14.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:14.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:14.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:14.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:15.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:15.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:15.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:15.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:15.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:15.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:16.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:16.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:16.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:16.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:17.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:17.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:17.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:17.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:17.759 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:56:17.751+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:17.757682+0000) 2019-10-18T01:56:18.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:18.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:18.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:18.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:18.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:56:18.629+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:18.630079+0000) 2019-10-18T01:56:19.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:19.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:19.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:19.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:20.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:20.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:20.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:20.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:20.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:20.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:21.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:21.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:21.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:21.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:21.752 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:56:21.743+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:21.749896+0000) 2019-10-18T01:56:22.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:22.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:22.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:22.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:23.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:23.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:23.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:23.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:23.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:56:23.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:23.175567+0000) 2019-10-18T01:56:24.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:24.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:24.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:24.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:25.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:25.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:25.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:25.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:25.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:25.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:26.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:26.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:27.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:27.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:27.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:27.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:27.760 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:56:27.751+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:27.757897+0000) 2019-10-18T01:56:28.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:28.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:28.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:28.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:28.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:56:28.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:28.630181+0000) 2019-10-18T01:56:29.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:29.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:29.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:29.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:30.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:30.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:30.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:30.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:30.790 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:30.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:31.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:31.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:31.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:31.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:31.752 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:56:31.743+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:31.750119+0000) 2019-10-18T01:56:32.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:32.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:32.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:32.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:33.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:32.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:33.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:32.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:33.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:56:33.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:33.175665+0000) 2019-10-18T01:56:34.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:34.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:34.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:34.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:35.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:35.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:35.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:35.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:35.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:35.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:36.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:35.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:36.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:35.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:37.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:37.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:37.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:37.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:37.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:56:37.751+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:37.758050+0000) 2019-10-18T01:56:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:38.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:38.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:38.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:56:38.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:38.630286+0000) 2019-10-18T01:56:39.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:39.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:39.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:39.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:40.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:40.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:40.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:40.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:40.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:40.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:41.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:41.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:41.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:41.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:41.752 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:56:41.747+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:41.750324+0000) 2019-10-18T01:56:41.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:41.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:41.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:41.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:42.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:42.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:42.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:42.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:43.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:56:43.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:43.175763+0000) 2019-10-18T01:56:43.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:43.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:43.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:43.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:44.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:44.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:44.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:44.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:45.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:45.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:45.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:45.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:45.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:45.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:47.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:47.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:47.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:47.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:47.760 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:56:47.755+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:47.758238+0000) 2019-10-18T01:56:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:48.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:48.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:48.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:56:48.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:48.630389+0000) 2019-10-18T01:56:48.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:48.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:48.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:48.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:49.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:49.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:49.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:49.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:50.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:50.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:50.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:50.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:50.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:50.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:51.752 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:56:51.747+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:51.750535+0000) 2019-10-18T01:56:51.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:51.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:51.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:51.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:52.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:52.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:52.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:52.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:53.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:56:53.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:53.175861+0000) 2019-10-18T01:56:53.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:53.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:53.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:53.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:54.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:54.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:54.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:54.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:55.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:56:55.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:56:55.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:55.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:55.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:55.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:56.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:56.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:56.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:56.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:57.760 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:56:57.755+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:57.758388+0000) 2019-10-18T01:56:58.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:57.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:58.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:57.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:58.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:56:58.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:56:58.630493+0000) 2019-10-18T01:56:58.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:58.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:58.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:58.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:56:59.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:59.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:56:59.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:56:59.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:00.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:00.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:00.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:00.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:46:24.777340+0000 (oldest deadline 2019-10-18T01:46:44.777340+0000) 2019-10-18T01:57:00.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:00.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:01.752 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:57:01.747+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:01.750718+0000) 2019-10-18T01:57:01.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:01.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:02.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:02.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:03.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:57:03.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:03.175954+0000) 2019-10-18T01:57:03.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:03.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:04.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:04.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:05.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:05.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:05.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:05.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:06.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:06.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:07.760 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:57:07.755+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:07.758568+0000) 2019-10-18T01:57:07.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:07.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:08.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:57:08.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:08.630589+0000) 2019-10-18T01:57:08.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:08.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:09.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:09.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:10.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:10.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:10.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:10.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:11.753 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:57:11.747+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:11.750934+0000) 2019-10-18T01:57:11.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:11.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:12.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:12.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:13.177 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:57:13.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:13.176051+0000) 2019-10-18T01:57:13.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:13.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:14.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:14.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:15.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:15.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:15.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:15.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:16.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:16.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:17.761 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:57:17.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:17.758750+0000) 2019-10-18T01:57:17.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:17.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:18.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:57:18.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:18.630694+0000) 2019-10-18T01:57:18.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:18.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:19.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:19.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:20.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:20.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:20.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:20.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:21.753 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:57:21.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:21.751146+0000) 2019-10-18T01:57:21.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:21.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:22.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:22.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:23.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:57:23.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:23.176152+0000) 2019-10-18T01:57:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:23.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:23.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:24.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:24.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:24.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:24.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:25.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:25.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:25.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:25.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:25.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:26.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:26.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:26.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:26.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:27.761 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:57:27.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:27.758927+0000) 2019-10-18T01:57:27.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:27.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:27.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:27.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:28.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:57:28.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:28.630797+0000) 2019-10-18T01:57:28.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:28.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:28.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:28.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:29.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:29.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:29.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:29.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:30.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:30.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:31.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:30.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:31.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:30.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:31.753 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:57:31.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:31.751333+0000) 2019-10-18T01:57:32.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:32.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:32.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:32.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:33.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:33.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:33.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:33.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:33.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:57:33.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:33.176254+0000) 2019-10-18T01:57:33.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:33.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:33.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:33.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:34.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:34.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:34.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:34.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:35.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:35.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:35.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:35.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:35.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:35.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:36.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:36.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:36.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:36.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:37.761 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:57:37.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:37.759108+0000) 2019-10-18T01:57:37.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:37.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:37.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:37.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:38.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:57:38.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:38.630906+0000) 2019-10-18T01:57:39.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:39.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:39.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:39.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:40.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:40.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:40.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:40.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:40.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:40.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:41.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:41.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:41.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:41.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:41.753 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:57:41.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:41.751521+0000) 2019-10-18T01:57:42.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:42.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:42.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:42.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:43.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:43.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:43.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:57:43.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:43.176352+0000) 2019-10-18T01:57:44.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:44.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:44.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:44.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:45.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:45.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:45.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:45.803 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:45.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:46.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:46.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:46.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:46.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:47.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:47.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:47.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:47.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:47.761 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:57:47.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:47.759290+0000) 2019-10-18T01:57:48.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:48.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:48.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:48.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:48.632 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:57:48.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:48.631016+0000) 2019-10-18T01:57:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:49.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:49.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:50.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:50.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:50.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:50.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:50.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:50.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:51.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:51.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:51.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:51.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:51.753 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:57:51.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:51.751734+0000) 2019-10-18T01:57:52.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:52.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:52.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:52.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:53.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:53.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:53.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:53.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:53.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:57:53.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:53.176450+0000) 2019-10-18T01:57:54.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:54.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:54.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:54.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:55.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:55.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:55.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:55.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:55.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:57:55.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:57:56.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:56.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:56.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:56.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:57.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:57.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:57.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:57.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:57.761 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:57:57.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:57.759473+0000) 2019-10-18T01:57:58.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:58.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:58.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:58.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:57:58.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:57:58.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:57:58.631123+0000) 2019-10-18T01:57:59.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:59.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:57:59.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:57:59.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:00.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:00.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:00.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:00.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:00.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:00.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:01.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:01.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:01.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:01.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:01.754 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:58:01.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:01.751928+0000) 2019-10-18T01:58:02.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:02.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:02.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:02.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:03.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:58:03.170+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:03.176551+0000) 2019-10-18T01:58:03.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:03.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:03.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:03.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:04.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:04.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:04.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:04.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:05.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:05.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:05.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:05.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:05.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:05.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:06.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:06.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:06.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:06.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:07.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:07.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:07.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:07.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:07.761 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:58:07.754+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:07.759660+0000) 2019-10-18T01:58:08.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:08.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:08.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:08.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:08.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:58:08.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:08.631245+0000) 2019-10-18T01:58:09.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:09.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:09.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:09.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:10.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:10.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:10.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:10.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:10.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:10.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:11.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:11.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:11.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:11.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:11.754 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:58:11.746+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:11.752118+0000) 2019-10-18T01:58:12.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:12.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:12.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:12.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:13.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:58:13.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:13.176648+0000) 2019-10-18T01:58:13.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:13.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:13.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:13.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:14.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:14.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:14.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:14.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:15.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:15.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:15.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:15.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:15.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:15.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:16.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:16.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:16.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:16.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:17.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:17.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:17.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:17.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:17.762 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:58:17.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:17.759840+0000) 2019-10-18T01:58:18.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:18.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:18.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:18.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:18.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:58:18.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:18.631353+0000) 2019-10-18T01:58:19.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:19.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:19.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:19.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:20.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:20.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:20.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:20.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:20.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:20.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:21.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:21.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:21.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:21.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:21.754 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:58:21.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:21.752310+0000) 2019-10-18T01:58:22.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:22.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:22.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:22.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:23.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:58:23.169+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:23.176749+0000) 2019-10-18T01:58:23.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:23.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:23.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:23.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:24.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:24.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:24.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:24.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:25.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:25.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:25.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:25.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:25.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:25.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:26.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:26.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:26.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:26.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:27.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:27.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:27.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:27.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:27.762 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:58:27.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:27.760027+0000) 2019-10-18T01:58:28.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:28.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:28.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:28.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:28.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:58:28.629+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:28.631458+0000) 2019-10-18T01:58:29.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:29.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:29.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:29.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:30.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:30.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:30.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:30.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:30.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:30.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:31.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:31.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:31.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:31.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:31.754 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:58:31.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:31.752501+0000) 2019-10-18T01:58:32.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:32.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:32.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:32.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:33.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:33.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:33.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:33.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:33.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:58:33.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:33.176847+0000) 2019-10-18T01:58:34.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:34.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:34.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:34.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:35.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:35.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:35.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:35.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:35.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:35.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:36.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:36.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:36.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:36.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:37.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:37.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:37.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:37.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:37.762 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:58:37.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:37.760205+0000) 2019-10-18T01:58:38.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:38.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:38.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:38.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:38.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:58:38.629+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:38.631562+0000) 2019-10-18T01:58:39.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:39.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:39.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:39.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:40.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:40.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:40.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:40.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:40.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:40.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:41.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:41.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:41.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:41.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:41.754 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:58:41.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:41.752684+0000) 2019-10-18T01:58:42.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:42.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:42.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:42.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:43.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:58:43.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:43.176947+0000) 2019-10-18T01:58:43.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:43.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:43.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:43.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:44.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:44.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:44.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:44.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:45.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:45.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:45.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:45.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:45.814 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:45.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:46.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:46.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:46.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:46.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:47.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:47.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:47.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:47.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:47.762 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:58:47.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:47.760386+0000) 2019-10-18T01:58:48.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:48.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:48.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:48.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:48.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:58:48.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:48.631669+0000) 2019-10-18T01:58:49.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:49.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:49.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:49.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:50.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:50.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:50.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:50.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:50.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:50.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:51.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:51.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:51.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:51.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:51.754 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:58:51.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:51.752870+0000) 2019-10-18T01:58:52.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:52.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:52.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:52.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:53.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:58:53.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:53.177046+0000) 2019-10-18T01:58:53.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:53.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:53.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:53.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:54.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:54.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:54.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:54.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:55.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:55.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:55.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:55.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:55.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:58:55.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:58:56.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:56.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:56.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:56.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:57.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:57.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:57.762 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:58:57.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:57.760572+0000) 2019-10-18T01:58:58.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:58.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:58.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:58.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:58:58.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:58:58.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:58:58.631776+0000) 2019-10-18T01:58:59.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:59.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:58:59.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:58:59.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:00.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:00.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:00.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:00.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:00.816 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:00.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:01.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:01.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:01.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:01.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:01.755 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:59:01.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:01.753058+0000) 2019-10-18T01:59:02.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:02.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:02.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:02.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:03.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:59:03.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:03.177146+0000) 2019-10-18T01:59:03.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:03.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:03.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:03.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:04.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:04.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:04.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:04.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:05.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:05.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:05.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:05.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:05.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:05.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:06.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:06.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:06.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:06.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:07.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:07.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:07.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:07.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:07.762 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:59:07.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:07.760754+0000) 2019-10-18T01:59:08.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:08.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:08.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:08.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:08.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:59:08.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:08.631884+0000) 2019-10-18T01:59:09.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:09.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:09.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:09.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:10.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:10.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:10.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:10.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:10.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:10.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:11.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:11.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:11.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:11.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:11.755 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:59:11.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:11.753247+0000) 2019-10-18T01:59:12.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:12.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:12.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:12.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:13.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:13.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:13.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:13.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:13.178 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:59:13.172+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:13.177246+0000) 2019-10-18T01:59:14.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:14.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:14.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:14.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:15.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:15.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:15.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:15.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:15.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:15.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:16.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:16.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:16.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:16.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:17.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:17.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:17.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:17.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:17.763 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:59:17.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:17.760937+0000) 2019-10-18T01:59:18.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:18.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:18.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:18.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:18.633 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:59:18.628+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:18.631988+0000) 2019-10-18T01:59:19.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:19.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:19.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:19.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:20.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:20.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:20.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:20.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:20.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:20.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:21.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:21.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:21.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:21.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:21.755 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:59:21.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:21.753423+0000) 2019-10-18T01:59:22.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:22.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:22.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:22.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:23.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:23.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:23.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:23.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:23.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:59:23.171+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:23.177343+0000) 2019-10-18T01:59:24.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:24.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:24.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:24.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:25.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:25.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:25.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:25.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:25.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:25.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:25.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:25.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:25.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:25.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:26.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:26.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:26.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:26.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:27.763 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:59:27.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:27.761120+0000) 2019-10-18T01:59:27.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:27.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:27.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:27.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:28.635 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:59:28.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:28.632089+0000) 2019-10-18T01:59:28.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:28.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:28.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:28.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:29.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:29.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:29.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:29.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:30.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:30.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:31.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:31.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:31.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:31.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:31.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:59:31.749+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:31.753610+0000) 2019-10-18T01:59:32.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:32.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:32.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:33.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:33.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:33.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:33.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:33.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:59:33.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:33.177442+0000) 2019-10-18T01:59:34.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:34.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:34.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:34.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:35.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:35.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:35.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:35.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:35.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:35.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:36.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:36.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:36.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:36.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:37.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:37.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:37.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:37.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:37.763 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:59:37.757+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:37.761307+0000) 2019-10-18T01:59:38.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:38.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:38.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:38.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:38.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:59:38.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:38.632191+0000) 2019-10-18T01:59:39.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:39.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:39.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:39.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:40.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:40.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:40.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:40.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:40.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:40.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:41.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:41.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:41.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:41.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:41.755 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:59:41.748+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:41.753793+0000) 2019-10-18T01:59:42.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:42.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:42.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:42.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:43.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:43.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:43.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:43.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:43.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:59:43.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:43.177550+0000) 2019-10-18T01:59:44.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:44.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:44.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:44.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:45.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:45.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:45.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:45.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:45.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:45.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:46.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:46.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:46.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:46.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:47.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:47.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:47.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:47.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:47.763 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:59:47.756+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:47.761544+0000) 2019-10-18T01:59:48.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:48.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:48.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:48.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:48.635 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:59:48.627+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:48.632298+0000) 2019-10-18T01:59:49.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:49.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:49.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:49.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:50.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:50.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:50.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:50.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:50.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:50.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:51.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:51.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:51.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:51.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:51.756 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T01:59:51.748+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:51.753983+0000) 2019-10-18T01:59:52.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:52.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:52.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:52.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:53.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:53.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:53.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:53.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:53.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T01:59:53.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:53.177645+0000) 2019-10-18T01:59:54.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:54.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:54.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:54.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:55.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:55.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:55.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:55.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:55.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T01:59:55.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T01:59:56.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:56.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:56.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:56.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:57.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:57.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:57.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:57.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:57.763 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T01:59:57.756+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:57.761730+0000) 2019-10-18T01:59:58.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:58.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:58.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:58.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T01:59:58.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T01:59:58.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T00:59:58.632399+0000) 2019-10-18T01:59:59.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:59.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T01:59:59.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T01:59:59.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:00.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:00.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:00.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:00.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:00.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:00.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:01.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:01.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:01.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:01.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:01.756 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:00:01.752+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:01.754174+0000) 2019-10-18T02:00:01.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:01.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:01.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:01.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:02.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:02.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:02.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:02.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:03.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:00:03.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:03.177744+0000) 2019-10-18T02:00:03.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:03.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:03.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:03.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:04.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:04.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:05.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:05.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:05.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:05.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:05.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:05.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:06.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:06.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:06.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:06.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:07.764 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:00:07.756+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:07.761918+0000) 2019-10-18T02:00:07.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:07.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:07.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:07.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:08.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:08.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:08.632494+0000) 2019-10-18T02:00:08.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:08.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:08.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:08.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:09.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:09.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:09.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:09.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:10.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:10.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:10.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:10.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:10.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:10.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:11.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:00:11.752+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:11.754411+0000) 2019-10-18T02:00:11.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:11.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:11.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:11.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:12.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:12.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:12.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:12.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:13.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:00:13.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:13.177842+0000) 2019-10-18T02:00:13.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:13.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:13.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:13.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:14.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:14.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:14.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:14.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:15.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:15.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:15.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:15.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:16.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:16.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:16.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:16.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:17.764 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:00:17.756+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:17.762101+0000) 2019-10-18T02:00:17.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:17.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:17.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:17.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:18.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:18.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:18.632597+0000) 2019-10-18T02:00:18.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:18.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:18.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:18.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:19.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:19.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:19.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:19.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:20.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:20.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:20.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:20.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:20.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:20.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:21.756 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:00:21.752+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:21.754616+0000) 2019-10-18T02:00:21.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:21.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:21.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:21.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:22.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:22.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:22.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:22.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:23.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:00:23.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:23.177945+0000) 2019-10-18T02:00:23.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:23.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:23.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:23.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:24.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:24.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:24.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:24.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:25.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:25.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:25.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:25.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:25.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:25.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:26.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:26.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:26.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:26.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:27.764 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:00:27.760+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:27.762277+0000) 2019-10-18T02:00:27.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:27.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:27.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:27.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:28.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:28.626+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:28.632694+0000) 2019-10-18T02:00:28.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:28.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:28.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:28.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:29.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:29.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:29.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:29.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:30.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:30.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:30.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:30.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:31.756 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:00:31.752+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:31.754798+0000) 2019-10-18T02:00:31.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:31.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:31.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:31.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:32.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:32.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:32.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:32.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:33.179 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:00:33.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:33.178041+0000) 2019-10-18T02:00:33.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:33.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:33.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:33.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:34.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:34.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:34.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:34.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:35.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:35.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:35.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:35.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:35.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:35.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:36.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:36.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:36.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:36.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:37.767 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:00:37.760+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:37.762447+0000) 2019-10-18T02:00:37.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:37.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:37.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:37.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:38.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:38.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:38.632798+0000) 2019-10-18T02:00:38.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:38.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:38.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:38.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:39.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:39.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:39.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:39.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:40.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:40.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:40.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:40.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:40.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:40.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:41.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:00:41.752+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:41.754982+0000) 2019-10-18T02:00:41.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:41.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:41.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:41.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:42.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:42.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:42.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:42.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:43.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:00:43.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:43.178142+0000) 2019-10-18T02:00:43.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:43.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:43.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:43.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:44.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:44.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:44.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:44.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:45.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:45.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:45.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:45.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:45.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:45.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:46.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:46.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:46.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:46.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:47.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:00:47.760+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:47.762639+0000) 2019-10-18T02:00:47.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:47.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:47.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:47.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:48.634 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:48.625+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:48.632897+0000) 2019-10-18T02:00:48.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:48.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:48.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:48.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:49.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:49.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:49.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:49.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:50.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:50.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:50.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:50.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:50.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:50.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:51.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:00:51.752+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:51.755195+0000) 2019-10-18T02:00:51.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:51.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:51.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:51.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:52.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:52.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:52.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:52.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:53.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:00:53.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:53.178244+0000) 2019-10-18T02:00:53.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:53.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:53.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:53.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:54.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:54.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:54.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:54.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:55.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:00:55.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:00:55.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:55.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:55.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:55.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:56.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:56.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:56.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:56.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:57.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:00:57.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:57.762831+0000) 2019-10-18T02:00:57.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:57.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:57.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:57.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:58.635 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:58.629+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:58.632992+0000) 2019-10-18T02:00:58.635 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:00:58.633+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:00:58.634346+0000) 2019-10-18T02:00:58.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:58.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:58.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:58.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:00:59.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:59.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:00:59.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:00:59.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:00.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:00.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:00.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:00.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:00.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:00.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:01.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:01:01.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:01.755376+0000) 2019-10-18T02:01:01.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:01.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:01.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:01.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:02.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:02.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:02.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:02.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:03.134 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:01:03.129+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:03.133189+0000) 2019-10-18T02:01:03.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:01:03.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:03.178341+0000) 2019-10-18T02:01:03.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:03.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:03.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:03.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:04.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:04.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:04.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:04.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:05.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:05.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:05.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:05.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:05.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:05.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:06.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:06.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:06.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:06.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:07.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:01:07.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:07.763043+0000) 2019-10-18T02:01:07.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:07.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:07.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:07.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:08.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:08.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:08.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:08.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:09.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:09.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:09.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:09.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:10.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:10.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:10.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:10.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:10.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:10.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:11.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:01:11.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:11.755560+0000) 2019-10-18T02:01:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:11.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:11.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:12.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:12.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:12.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:12.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:13.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:01:13.128+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:13.133288+0000) 2019-10-18T02:01:13.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:01:13.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:13.178442+0000) 2019-10-18T02:01:13.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:13.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:13.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:13.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:14.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:14.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:14.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:14.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:15.839 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:15.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:15.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:15.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:15.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:15.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:16.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:16.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:16.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:16.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:17.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:01:17.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:17.763192+0000) 2019-10-18T02:01:17.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:17.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:17.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:17.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:18.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:18.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:18.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:18.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:19.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:19.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:19.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:19.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:20.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:20.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:20.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:20.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:20.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:20.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:21.757 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:01:21.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:21.755755+0000) 2019-10-18T02:01:21.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:21.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:21.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:21.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:22.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:22.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:22.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:22.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:23.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:01:23.128+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:23.133387+0000) 2019-10-18T02:01:23.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:01:23.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:23.178538+0000) 2019-10-18T02:01:23.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:23.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:23.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:23.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:24.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:24.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:24.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:24.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:25.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:25.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:25.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:25.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:25.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:25.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:26.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:26.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:26.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:26.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:27.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:01:27.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:27.763372+0000) 2019-10-18T02:01:27.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:27.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:27.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:27.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:28.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:28.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:28.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:28.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:29.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:29.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:29.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:29.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:30.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:30.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:30.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:30.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:30.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:30.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:31.758 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:01:31.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:31.755938+0000) 2019-10-18T02:01:31.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:31.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:31.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:31.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:32.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:32.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:32.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:32.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:33.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:01:33.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:33.133487+0000) 2019-10-18T02:01:33.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:01:33.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:33.178636+0000) 2019-10-18T02:01:33.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:33.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:33.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:33.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:34.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:34.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:34.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:34.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:35.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:35.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:35.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:35.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:35.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:35.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:36.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:36.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:36.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:36.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:37.765 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:01:37.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:37.763559+0000) 2019-10-18T02:01:37.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:37.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:37.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:37.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:38.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:38.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:38.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:38.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:39.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:39.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:39.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:39.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:40.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:40.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:40.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:40.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:40.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:40.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:41.758 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:01:41.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:41.756128+0000) 2019-10-18T02:01:41.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:41.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:41.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:41.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:42.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:42.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:42.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:42.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:43.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:01:43.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:43.133582+0000) 2019-10-18T02:01:43.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:01:43.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:43.178736+0000) 2019-10-18T02:01:43.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:43.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:43.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:43.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:44.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:44.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:44.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:44.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:45.845 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:45.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:45.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:45.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:45.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:45.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:47.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:46.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:47.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:46.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:47.766 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:01:47.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:47.763746+0000) 2019-10-18T02:01:48.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:48.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:48.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:48.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:49.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:49.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:49.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:49.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:50.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:50.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:50.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:50.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:50.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:50.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:50.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:50.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:50.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:50.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:51.758 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:01:51.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:51.756322+0000) 2019-10-18T02:01:52.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:52.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:52.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:52.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:52.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:52.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:52.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:52.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:53.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:01:53.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:53.133680+0000) 2019-10-18T02:01:53.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:01:53.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:53.178835+0000) 2019-10-18T02:01:54.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:54.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:54.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:54.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:54.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:54.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:54.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:54.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:55.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:01:55.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:01:56.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:56.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:56.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:56.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:57.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:57.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:57.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:57.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:57.766 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:01:57.759+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:01:57.763932+0000) 2019-10-18T02:01:58.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:58.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:58.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:58.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:58.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:58.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:58.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:58.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:01:59.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:59.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:01:59.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:01:59.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:00.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:00.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:00.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:00.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:00.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:00.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:01.758 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:02:01.751+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:01.756511+0000) 2019-10-18T02:02:01.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:01.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:01.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:01.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:02.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:02.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:02.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:02.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:03.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:02:03.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:03.133782+0000) 2019-10-18T02:02:03.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:02:03.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:03.178933+0000) 2019-10-18T02:02:03.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:03.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:03.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:03.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:04.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:04.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:04.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:04.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:05.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:05.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:05.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:05.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:05.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:05.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:06.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:06.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:06.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:06.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:07.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:02:07.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:07.764144+0000) 2019-10-18T02:02:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:08.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:08.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:08.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:08.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:09.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:08.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:10.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:10.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:10.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:10.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:10.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:10.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:11.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:11.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:11.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:11.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:11.758 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:02:11.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:11.756702+0000) 2019-10-18T02:02:12.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:12.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:12.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:12.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:13.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:13.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:13.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:13.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:13.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:02:13.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:13.133875+0000) 2019-10-18T02:02:13.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:02:13.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:13.179033+0000) 2019-10-18T02:02:14.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:14.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:14.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:14.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:15.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:15.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:15.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:15.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:15.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:15.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:16.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:16.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:16.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:16.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:17.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:17.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:17.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:17.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:17.766 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:02:17.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:17.764330+0000) 2019-10-18T02:02:18.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:18.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:18.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:18.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:19.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:19.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:19.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:19.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:20.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:20.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:20.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:20.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:21.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:21.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:21.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:21.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:21.758 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:02:21.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:21.756889+0000) 2019-10-18T02:02:22.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:22.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:22.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:22.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:23.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:02:23.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:23.133966+0000) 2019-10-18T02:02:23.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:02:23.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:23.179131+0000) 2019-10-18T02:02:23.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:23.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:23.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:23.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:24.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:24.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:24.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:24.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:25.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:25.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:25.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:25.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:25.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:25.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:26.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:26.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:26.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:26.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:27.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:27.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:27.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:27.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:27.766 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:02:27.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:27.764551+0000) 2019-10-18T02:02:28.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:28.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:28.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:28.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:29.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:29.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:29.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:29.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:30.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:30.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:30.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:30.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:30.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:30.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:31.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:31.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:31.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:31.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:31.759 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:02:31.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:31.757078+0000) 2019-10-18T02:02:32.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:32.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:32.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:32.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:33.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:02:33.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:33.134061+0000) 2019-10-18T02:02:33.180 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:02:33.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:33.179230+0000) 2019-10-18T02:02:33.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:33.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:33.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:33.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:34.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:34.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:34.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:34.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:35.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:35.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:35.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:35.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:35.853 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:35.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:36.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:36.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:36.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:36.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:37.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:37.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:37.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:37.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:37.766 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:02:37.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:37.764742+0000) 2019-10-18T02:02:38.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:38.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:38.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:38.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:39.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:39.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:39.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:39.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:40.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:40.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:40.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:40.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:40.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:40.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:41.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:41.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:41.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:41.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:41.759 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:02:41.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:41.757264+0000) 2019-10-18T02:02:42.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:42.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:42.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:42.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:43.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:02:43.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:43.134156+0000) 2019-10-18T02:02:43.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:02:43.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:43.179328+0000) 2019-10-18T02:02:43.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:43.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:43.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:43.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:44.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:44.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:44.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:44.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:45.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:45.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:45.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:45.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:45.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:45.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:46.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:46.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:46.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:46.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:47.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:47.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:47.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:47.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:47.766 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:02:47.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:47.764929+0000) 2019-10-18T02:02:48.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:48.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:48.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:48.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:49.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:49.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:49.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:49.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:50.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:50.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:50.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:50.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:50.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:50.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:51.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:51.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:51.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:51.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:51.759 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:02:51.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:51.757454+0000) 2019-10-18T02:02:52.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:52.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:52.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:52.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:53.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:02:53.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:53.134256+0000) 2019-10-18T02:02:53.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:53.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:53.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:53.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:53.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:02:53.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:53.179427+0000) 2019-10-18T02:02:54.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:54.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:54.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:54.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:55.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:55.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:55.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:55.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:55.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:02:55.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:02:56.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:56.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:56.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:56.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:57.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:57.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:57.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:57.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:57.767 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:02:57.758+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:02:57.765073+0000) 2019-10-18T02:02:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:58.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:58.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:02:59.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:59.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:02:59.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:02:59.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:00.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:00.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:00.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:00.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:00.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:00.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:01.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:01.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:01.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:01.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:01.759 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:03:01.750+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:01.757642+0000) 2019-10-18T02:03:02.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:02.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:02.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:02.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:03.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:03.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:03.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:03.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:03.135 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:03:03.129+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:03.134357+0000) 2019-10-18T02:03:03.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:03:03.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:03.179523+0000) 2019-10-18T02:03:04.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:04.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:04.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:04.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:05.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:05.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:05.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:05.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:05.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:05.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:06.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:06.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:06.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:06.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:07.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:07.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:07.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:07.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:07.767 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:03:07.762+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:07.765271+0000) 2019-10-18T02:03:08.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:08.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:08.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:08.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:09.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:09.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:09.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:09.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:10.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:10.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:10.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:10.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:10.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:10.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:11.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:11.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:11.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:11.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:11.759 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:03:11.754+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:11.757824+0000) 2019-10-18T02:03:12.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:12.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:12.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:12.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:13.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:03:13.129+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:13.134467+0000) 2019-10-18T02:03:13.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:03:13.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:13.179620+0000) 2019-10-18T02:03:13.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:13.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:13.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:13.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:14.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:14.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:14.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:14.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:15.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:15.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:15.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:15.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:15.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:15.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:16.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:16.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:16.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:16.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:17.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:17.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:17.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:17.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:17.767 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:03:17.762+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:17.765439+0000) 2019-10-18T02:03:18.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:18.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:18.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:18.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:19.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:19.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:19.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:19.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:20.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:20.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:20.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:20.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:20.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:20.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:21.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:21.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:21.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:21.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:21.760 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:03:21.753+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:21.758043+0000) 2019-10-18T02:03:22.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:22.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:22.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:22.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:23.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:23.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:23.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:23.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:23.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:03:23.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:23.134567+0000) 2019-10-18T02:03:23.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:03:23.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:23.179716+0000) 2019-10-18T02:03:24.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:24.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:24.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:24.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:25.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:25.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:25.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:25.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:25.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:25.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:26.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:26.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:26.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:26.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:27.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:27.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:27.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:27.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:27.767 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:03:27.761+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:27.765637+0000) 2019-10-18T02:03:28.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:28.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:28.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:28.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:29.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:29.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:29.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:29.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:30.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:30.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:30.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:30.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:30.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:30.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:31.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:31.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:31.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:31.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:31.760 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:03:31.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:31.758229+0000) 2019-10-18T02:03:32.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:32.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:32.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:32.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:33.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:33.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:33.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:33.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:33.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:03:33.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:33.134667+0000) 2019-10-18T02:03:33.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:03:33.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:33.179812+0000) 2019-10-18T02:03:34.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:34.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:34.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:34.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:34.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:34.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:34.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:34.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:35.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:35.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:35.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:35.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:35.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:35.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:36.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:36.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:36.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:36.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:37.767 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:03:37.761+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:37.765827+0000) 2019-10-18T02:03:37.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:37.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:37.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:37.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:38.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:38.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:38.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:38.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:39.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:39.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:39.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:39.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:40.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:40.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:40.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:40.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:40.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:40.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:41.760 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:03:41.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:41.758422+0000) 2019-10-18T02:03:41.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:41.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:41.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:41.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:42.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:42.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:42.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:42.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:43.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:03:43.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:43.134765+0000) 2019-10-18T02:03:43.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:03:43.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:43.179913+0000) 2019-10-18T02:03:43.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:43.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:43.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:43.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:45.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:45.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:45.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:45.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:45.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:45.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:46.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:46.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:46.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:46.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:47.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:47.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:47.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:47.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:47.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:03:47.761+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:47.766004+0000) 2019-10-18T02:03:48.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:48.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:48.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:48.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:49.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:49.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:49.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:49.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:50.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:50.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:50.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:50.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:50.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:51.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:51.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:51.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:51.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:51.760 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:03:51.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:51.758621+0000) 2019-10-18T02:03:52.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:52.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:52.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:52.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:53.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:03:53.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:53.134867+0000) 2019-10-18T02:03:53.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:53.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:53.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:53.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:53.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:03:53.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:53.180013+0000) 2019-10-18T02:03:54.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:54.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:54.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:54.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:55.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:55.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:55.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:55.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:55.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:03:55.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:03:56.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:56.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:56.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:56.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:57.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:57.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:57.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:57.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:57.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:03:57.765+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:03:57.766217+0000) 2019-10-18T02:03:58.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:58.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:58.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:58.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:03:59.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:59.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:03:59.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:03:59.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:00.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:00.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:00.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:00.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:00.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:00.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:01.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:01.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:01.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:01.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:01.765 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:04:01.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:01.758764+0000) 2019-10-18T02:04:02.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:02.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:02.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:02.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:03.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:04:03.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:03.134971+0000) 2019-10-18T02:04:03.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:04:03.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:03.180114+0000) 2019-10-18T02:04:03.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:03.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:03.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:03.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:04.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:04.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:04.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:04.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:05.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:05.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:05.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:05.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:05.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:05.864+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:06.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:06.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:06.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:06.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:07.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:07.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:07.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:07.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:07.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:04:07.765+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:07.766374+0000) 2019-10-18T02:04:08.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:08.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:08.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:08.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:09.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:09.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:09.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:09.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:10.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:10.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:10.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:10.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:10.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:10.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:11.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:11.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:11.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:11.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:11.761 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:04:11.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:11.758940+0000) 2019-10-18T02:04:12.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:12.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:12.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:12.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:13.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:04:13.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:13.135068+0000) 2019-10-18T02:04:13.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:13.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:13.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:13.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:13.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:04:13.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:13.180214+0000) 2019-10-18T02:04:14.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:14.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:14.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:14.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:15.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:15.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:15.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:15.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:15.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:15.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:16.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:16.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:17.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:17.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:17.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:17.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:17.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:04:17.765+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:17.766550+0000) 2019-10-18T02:04:18.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:18.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:18.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:18.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:19.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:19.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:19.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:19.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:20.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:20.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:20.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:20.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:20.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:20.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:21.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:21.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:21.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:21.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:21.761 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:04:21.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:21.759124+0000) 2019-10-18T02:04:22.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:22.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:22.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:22.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:23.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:04:23.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:23.135168+0000) 2019-10-18T02:04:23.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:23.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:23.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:23.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:23.181 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:04:23.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:23.180315+0000) 2019-10-18T02:04:24.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:24.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:24.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:24.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:25.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:25.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:25.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:25.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:25.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:25.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:26.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:26.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:27.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:27.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:27.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:27.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:27.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:04:27.765+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:27.766789+0000) 2019-10-18T02:04:28.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:28.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:28.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:28.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:29.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:29.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:29.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:29.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:30.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:30.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:30.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:30.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:30.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:30.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:31.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:31.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:31.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:31.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:31.761 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:04:31.757+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:31.759282+0000) 2019-10-18T02:04:32.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:32.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:32.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:32.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:33.136 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:04:33.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:33.135268+0000) 2019-10-18T02:04:33.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:04:33.175+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:33.180414+0000) 2019-10-18T02:04:33.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:33.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:33.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:33.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:34.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:34.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:34.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:34.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:35.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:35.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:35.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:35.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:35.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:35.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:36.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:36.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:36.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:36.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:37.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:37.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:37.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:37.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:37.768 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:04:37.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:37.766970+0000) 2019-10-18T02:04:38.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:38.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:38.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:38.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:39.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:39.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:39.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:39.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:40.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:40.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:40.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:40.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:40.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:40.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:41.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:41.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:41.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:41.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:41.761 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:04:41.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:41.759498+0000) 2019-10-18T02:04:42.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:42.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:42.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:42.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:43.138 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:04:43.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:43.135366+0000) 2019-10-18T02:04:43.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:04:43.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:43.180516+0000) 2019-10-18T02:04:43.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:43.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:43.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:43.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:44.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:44.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:44.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:44.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:45.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:45.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:45.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:45.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:45.875 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:45.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:46.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:46.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:46.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:46.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:47.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:47.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:47.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:47.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:47.769 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:04:47.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:47.767145+0000) 2019-10-18T02:04:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:48.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:48.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:49.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:49.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:49.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:49.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:50.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:50.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:50.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:50.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:50.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:50.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:51.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:51.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:51.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:51.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:51.761 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:04:51.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:51.759695+0000) 2019-10-18T02:04:52.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:52.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:52.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:52.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:53.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:04:53.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:53.135467+0000) 2019-10-18T02:04:53.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:04:53.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:53.180609+0000) 2019-10-18T02:04:53.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:53.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:53.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:53.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:54.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:54.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:54.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:54.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:55.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:55.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:55.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:55.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:55.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:04:55.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:04:56.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:56.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:56.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:56.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:57.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:57.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:57.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:57.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:57.769 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:04:57.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:04:57.767337+0000) 2019-10-18T02:04:58.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:58.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:58.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:58.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:04:59.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:59.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:04:59.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:04:59.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:00.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:00.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:00.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:00.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:00.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:00.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:01.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:01.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:01.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:01.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:01.762 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:05:01.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:01.759878+0000) 2019-10-18T02:05:02.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:02.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:02.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:02.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:03.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:05:03.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:03.135575+0000) 2019-10-18T02:05:03.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:05:03.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:03.180704+0000) 2019-10-18T02:05:03.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:03.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:03.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:03.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:04.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:04.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:04.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:04.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:05.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:05.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:05.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:05.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:05.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:05.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:06.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:06.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:06.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:06.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:07.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:07.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:07.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:07.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:07.769 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:05:07.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:07.767517+0000) 2019-10-18T02:05:08.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:08.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:08.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:09.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:09.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:09.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:09.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:10.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:10.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:10.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:10.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:10.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:10.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:11.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:11.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:11.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:11.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:11.762 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:05:11.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:11.760067+0000) 2019-10-18T02:05:12.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:12.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:12.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:12.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:13.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:05:13.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:13.135672+0000) 2019-10-18T02:05:13.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:05:13.174+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:13.180798+0000) 2019-10-18T02:05:13.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:13.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:13.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:13.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:14.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:14.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:14.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:14.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:15.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:15.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:15.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:15.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:15.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:15.874+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:16.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:16.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:16.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:16.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:17.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:17.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:17.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:17.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:17.769 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:05:17.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:17.767696+0000) 2019-10-18T02:05:18.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:18.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:18.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:18.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:19.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:19.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:19.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:19.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:20.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:20.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:20.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:20.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:20.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:20.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:21.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:21.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:21.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:21.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:21.762 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:05:21.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:21.760247+0000) 2019-10-18T02:05:22.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:22.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:22.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:22.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:23.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:05:23.129+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:23.135768+0000) 2019-10-18T02:05:23.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:05:23.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:23.180893+0000) 2019-10-18T02:05:23.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:23.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:23.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:23.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:24.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:24.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:24.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:24.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:25.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:25.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:25.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:25.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:25.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:25.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:26.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:26.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:26.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:26.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:27.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:27.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:27.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:27.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:27.769 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:05:27.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:27.767873+0000) 2019-10-18T02:05:28.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:28.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:28.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:28.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:29.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:29.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:29.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:29.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:30.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:30.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:30.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:30.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:30.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:30.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:31.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:31.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:31.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:31.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:31.762 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:05:31.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:31.760441+0000) 2019-10-18T02:05:32.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:32.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:32.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:32.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:33.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:05:33.129+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:33.135866+0000) 2019-10-18T02:05:33.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:05:33.173+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:33.180993+0000) 2019-10-18T02:05:33.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:33.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:33.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:33.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:34.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:34.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:34.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:34.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:35.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:35.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:35.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:35.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:35.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:35.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:36.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:36.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:36.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:36.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:37.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:37.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:37.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:37.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:37.769 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:05:37.764+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:37.768057+0000) 2019-10-18T02:05:38.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:38.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:38.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:38.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:39.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:39.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:39.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:39.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:40.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:40.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:40.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:40.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:40.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:40.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:41.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:41.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:41.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:41.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:41.762 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:05:41.756+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:41.760630+0000) 2019-10-18T02:05:42.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:42.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:42.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:42.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:43.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:05:43.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:43.135961+0000) 2019-10-18T02:05:43.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:05:43.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:43.181089+0000) 2019-10-18T02:05:43.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:43.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:43.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:43.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:44.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:44.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:44.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:44.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:45.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:45.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:45.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:45.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:45.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:45.881+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:46.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:46.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:46.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:46.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:47.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:47.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:47.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:47.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:47.770 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:05:47.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:47.768197+0000) 2019-10-18T02:05:48.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:48.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:48.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:48.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:49.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:49.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:49.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:49.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:50.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:50.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:50.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:50.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:50.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:50.885+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:51.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:51.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:51.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:51.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:51.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:05:51.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:51.760840+0000) 2019-10-18T02:05:52.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:52.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:52.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:52.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:53.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:05:53.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:53.136054+0000) 2019-10-18T02:05:53.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:05:53.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:53.181182+0000) 2019-10-18T02:05:53.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:53.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:53.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:53.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:54.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:54.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:54.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:54.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:55.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:55.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:55.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:55.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:55.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:05:55.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:05:56.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:56.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:56.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:56.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:57.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:57.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:57.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:57.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:57.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:05:57.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:05:57.768387+0000) 2019-10-18T02:05:58.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:58.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:58.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:58.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:05:59.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:59.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:05:59.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:05:59.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:00.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:00.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:00.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:00.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:00.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:00.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:01.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:01.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:01.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:01.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:01.763 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:06:01.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:01.761034+0000) 2019-10-18T02:06:02.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:02.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:02.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:02.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:03.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:06:03.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:03.136156+0000) 2019-10-18T02:06:03.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:06:03.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:03.181282+0000) 2019-10-18T02:06:03.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:03.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:03.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:03.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:04.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:04.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:04.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:04.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:05.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:05.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:05.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:05.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:05.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:05.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:06.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:06.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:06.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:06.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:07.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:07.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:07.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:07.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:07.770 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:06:07.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:07.768569+0000) 2019-10-18T02:06:08.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:08.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:08.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:08.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:09.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:09.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:09.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:09.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:10.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:10.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:10.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:10.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:10.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:10.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:11.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:11.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:11.764 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:06:11.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:11.761224+0000) 2019-10-18T02:06:12.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:12.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:12.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:12.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:13.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:06:13.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:13.136257+0000) 2019-10-18T02:06:13.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:06:13.176+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:13.181384+0000) 2019-10-18T02:06:13.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:13.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:13.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:13.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:14.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:14.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:14.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:14.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:15.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:15.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:15.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:15.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:15.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:15.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:16.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:16.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:16.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:16.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:17.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:17.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:17.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:17.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:17.770 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:06:17.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:17.768749+0000) 2019-10-18T02:06:18.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:18.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:18.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:18.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:19.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:19.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:19.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:19.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:20.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:20.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:20.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:20.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:20.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:20.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:21.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:21.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:21.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:21.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:21.763 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:06:21.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:21.761411+0000) 2019-10-18T02:06:22.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:22.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:22.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:22.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:23.137 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:06:23.132+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:23.136356+0000) 2019-10-18T02:06:23.182 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:06:23.180+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:23.181463+0000) 2019-10-18T02:06:23.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:23.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:23.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:23.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:24.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:24.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:24.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:24.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:25.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:25.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:25.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:25.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:25.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:25.888+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:26.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:26.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:26.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:26.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:27.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:27.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:27.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:27.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:27.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:06:27.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:27.768931+0000) 2019-10-18T02:06:28.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:28.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:28.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:28.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:29.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:29.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:29.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:29.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:30.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:30.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:30.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:30.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:30.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:30.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:31.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:31.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:31.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:31.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:31.763 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:06:31.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:31.761595+0000) 2019-10-18T02:06:32.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:32.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:32.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:32.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:33.138 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:06:33.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:33.136457+0000) 2019-10-18T02:06:33.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:06:33.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:33.181562+0000) 2019-10-18T02:06:33.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:33.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:33.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:33.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:34.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:34.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:34.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:34.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:35.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:35.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:35.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:35.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:35.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:35.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:36.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:36.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:36.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:36.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:37.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:37.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:37.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:37.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:37.770 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:06:37.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:37.769113+0000) 2019-10-18T02:06:38.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:38.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:38.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:38.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:39.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:39.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:39.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:39.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:40.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:40.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:40.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:40.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:40.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:40.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:41.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:41.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:41.763 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:06:41.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:41.761776+0000) 2019-10-18T02:06:42.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:42.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:42.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:42.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:43.138 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:06:43.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:43.136557+0000) 2019-10-18T02:06:43.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:06:43.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:43.181660+0000) 2019-10-18T02:06:43.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:43.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:43.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:43.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:44.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:44.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:44.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:44.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:45.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:45.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:45.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:45.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:45.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:45.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:46.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:46.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:46.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:46.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:47.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:47.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:47.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:47.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:47.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:06:47.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:47.769293+0000) 2019-10-18T02:06:48.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:48.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:48.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:48.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:49.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:49.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:49.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:49.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:50.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:50.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:50.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:50.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:50.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:50.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:51.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:51.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:51.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:51.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:51.763 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:06:51.755+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:51.761970+0000) 2019-10-18T02:06:52.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:52.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:52.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:52.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:53.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:53.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:53.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:53.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:53.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:06:53.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:53.136656+0000) 2019-10-18T02:06:53.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:06:53.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:53.181759+0000) 2019-10-18T02:06:54.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:54.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:54.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:54.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:55.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:55.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:55.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:55.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:55.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:06:55.891+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:06:56.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:56.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:56.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:56.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:57.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:57.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:57.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:57.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:57.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:06:57.763+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:06:57.769471+0000) 2019-10-18T02:06:58.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:58.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:58.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:58.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:06:59.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:59.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:06:59.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:06:59.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:00.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:00.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:00.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:00.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:00.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:00.895+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:01.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:01.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:01.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:01.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:01.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:07:01.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:01.762162+0000) 2019-10-18T02:07:02.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:02.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:02.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:02.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:03.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:03.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:03.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:03.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:03.138 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:03.131+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:03.136757+0000) 2019-10-18T02:07:03.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:07:03.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:03.181854+0000) 2019-10-18T02:07:04.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:04.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:04.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:04.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:05.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:05.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:05.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:05.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:05.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:05.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:06.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:06.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:06.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:06.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:07.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:07.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:07.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:07.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:07.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:07:07.762+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:07.769651+0000) 2019-10-18T02:07:08.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:08.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:08.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:08.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:09.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:09.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:09.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:09.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:10.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:10.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:10.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:10.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:10.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:10.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:11.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:11.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:11.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:11.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:11.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:07:11.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:11.762372+0000) 2019-10-18T02:07:12.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:12.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:12.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:12.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:13.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:13.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:13.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:13.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:13.138 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:13.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:13.136857+0000) 2019-10-18T02:07:13.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:07:13.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:13.181954+0000) 2019-10-18T02:07:14.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:14.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:14.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:14.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:15.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:15.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:15.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:15.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:15.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:15.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:16.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:15.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:16.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:15.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:16.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:16.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:16.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:16.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:17.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:07:17.762+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:17.769834+0000) 2019-10-18T02:07:17.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:17.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:17.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:17.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:18.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:18.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:18.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:18.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:19.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:19.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:19.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:19.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:20.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:20.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:20.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:20.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:20.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:20.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:21.764 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:07:21.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:21.762553+0000) 2019-10-18T02:07:21.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:21.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:21.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:21.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:22.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:22.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:22.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:22.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:23.138 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:23.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:23.136950+0000) 2019-10-18T02:07:23.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:07:23.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:23.182055+0000) 2019-10-18T02:07:23.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:23.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:23.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:23.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:24.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:24.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:24.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:24.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:25.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:25.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:25.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:25.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:25.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:25.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:26.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:26.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:26.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:26.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:27.771 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:07:27.762+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:27.770027+0000) 2019-10-18T02:07:27.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:27.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:27.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:27.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:28.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:28.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:28.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:28.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:29.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:29.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:29.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:29.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:30.904 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:30.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:30.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:30.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:31.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:07:31.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:31.762745+0000) 2019-10-18T02:07:31.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:31.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:31.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:31.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:32.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:32.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:32.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:32.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:33.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:33.130+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:33.137046+0000) 2019-10-18T02:07:33.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:33.134+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:33.138343+0000) 2019-10-18T02:07:33.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:07:33.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:33.182152+0000) 2019-10-18T02:07:33.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:33.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:33.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:33.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:35.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:35.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:35.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:35.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:35.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:35.898+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:36.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:36.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:36.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:36.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:37.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:37.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:37.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:37.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:37.639 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:37.630+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:37.637246+0000) 2019-10-18T02:07:37.772 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:07:37.766+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:37.770214+0000) 2019-10-18T02:07:38.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:38.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:38.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:38.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:39.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:39.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:39.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:39.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:40.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:40.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T01:57:03.310914+0000 (oldest deadline 2019-10-18T01:57:23.310914+0000) 2019-10-18T02:07:40.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:40.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:40.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:40.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:41.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:41.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:41.764 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:07:41.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:41.762926+0000) 2019-10-18T02:07:42.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:42.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:43.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:43.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:43.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:07:43.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:43.182255+0000) 2019-10-18T02:07:44.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:44.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:45.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:45.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:45.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:46.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:46.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:47.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:47.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:47.638 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:47.629+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:47.637351+0000) 2019-10-18T02:07:47.772 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:07:47.766+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:47.770385+0000) 2019-10-18T02:07:48.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:48.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:49.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:49.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:50.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:50.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:50.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:50.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:51.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:51.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:51.764 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:07:51.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:51.763105+0000) 2019-10-18T02:07:52.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:52.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:53.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:53.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:53.183 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:07:53.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:53.182352+0000) 2019-10-18T02:07:54.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:54.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:55.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:55.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:55.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:07:55.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:07:56.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:56.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:57.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:57.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:57.639 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:07:57.633+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:57.637461+0000) 2019-10-18T02:07:57.772 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:07:57.766+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:07:57.770537+0000) 2019-10-18T02:07:58.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:58.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:07:59.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:07:59.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:00.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:00.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:00.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:00.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:01.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:01.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:01.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:01.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:01.765 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:08:01.758+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:01.763322+0000) 2019-10-18T02:08:02.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:02.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:02.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:02.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:03.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:03.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:03.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:03.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:03.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:08:03.181+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:03.182449+0000) 2019-10-18T02:08:04.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:04.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:04.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:04.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:05.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:05.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:05.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:05.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:05.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:05.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:06.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:06.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:06.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:06.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:07.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:07.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:07.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:07.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:07.639 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:07.637+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:07.637553+0000) 2019-10-18T02:08:07.772 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:08:07.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:07.770718+0000) 2019-10-18T02:08:08.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:08.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:08.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:08.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:09.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:09.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:09.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:09.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:10.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:10.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:10.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:10.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:10.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:10.909+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:11.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:11.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:11.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:11.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:11.765 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:08:11.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:11.763514+0000) 2019-10-18T02:08:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:12.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:12.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:13.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:13.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:13.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:13.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:13.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:08:13.181+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:13.182550+0000) 2019-10-18T02:08:14.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:14.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:14.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:14.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:15.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:15.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:15.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:15.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:15.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:15.909+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:16.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:16.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:16.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:16.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:17.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:17.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:17.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:17.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:17.639 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:17.636+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:17.637655+0000) 2019-10-18T02:08:17.639 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:17.636+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:17.638516+0000) 2019-10-18T02:08:17.773 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:08:17.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:17.770933+0000) 2019-10-18T02:08:18.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:18.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:18.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:18.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:19.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:19.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:19.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:19.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:20.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:20.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:20.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:20.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:20.913 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:20.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:20.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:20.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:20.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:20.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:21.765 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:08:21.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:21.763702+0000) 2019-10-18T02:08:22.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:22.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:22.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:22.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:22.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:22.137861+0000) 2019-10-18T02:08:23.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:23.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:23.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:23.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:23.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:08:23.180+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:23.182643+0000) 2019-10-18T02:08:24.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:24.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:24.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:24.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:25.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:25.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:25.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:25.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:25.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:25.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:25.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:25.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:25.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:25.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:26.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:26.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:26.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:26.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:27.773 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:08:27.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:27.771142+0000) 2019-10-18T02:08:28.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:27.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:28.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:27.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:28.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:28.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:28.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:28.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:29.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:29.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:29.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:29.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:30.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:30.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:30.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:30.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:31.765 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:08:31.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:31.763849+0000) 2019-10-18T02:08:32.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:32.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:32.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:32.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:32.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:32.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:32.137968+0000) 2019-10-18T02:08:32.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:32.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:32.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:32.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:33.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:08:33.180+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:33.182738+0000) 2019-10-18T02:08:34.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:33.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:34.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:33.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:34.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:34.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:34.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:34.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:35.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:35.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:35.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:35.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:35.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:35.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:36.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:36.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:36.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:36.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:37.773 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:08:37.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:37.771318+0000) 2019-10-18T02:08:37.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:37.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:37.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:37.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:38.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:38.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:38.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:38.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:39.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:39.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:39.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:39.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:40.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:40.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:40.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:40.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:40.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:40.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:41.765 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:08:41.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:41.764064+0000) 2019-10-18T02:08:41.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:41.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:41.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:41.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:42.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:42.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:42.138078+0000) 2019-10-18T02:08:42.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:42.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:42.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:42.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:43.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:08:43.180+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:43.182841+0000) 2019-10-18T02:08:43.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:43.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:43.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:43.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:44.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:44.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:44.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:44.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:45.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:45.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:46.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:45.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:46.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:45.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:46.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:47.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:46.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:47.773 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:08:47.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:47.771533+0000) 2019-10-18T02:08:48.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:48.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:48.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:48.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:49.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:49.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:49.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:49.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:50.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:50.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:50.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:50.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:50.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:50.912+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:51.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:51.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:51.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:51.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:51.766 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:08:51.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:51.764222+0000) 2019-10-18T02:08:52.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:52.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:52.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:52.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:52.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:08:52.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:52.138187+0000) 2019-10-18T02:08:53.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:53.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:53.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:53.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:53.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:08:53.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:53.182940+0000) 2019-10-18T02:08:54.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:54.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:54.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:54.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:55.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:55.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:55.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:55.919 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:08:55.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:08:56.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:56.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:56.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:56.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:57.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:57.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:57.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:57.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:57.773 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:08:57.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:08:57.771725+0000) 2019-10-18T02:08:58.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:58.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:58.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:58.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:08:59.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:59.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:08:59.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:08:59.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:00.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:00.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:00.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:00.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:00.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:00.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:01.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:01.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:01.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:01.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:01.766 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:09:01.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:01.764413+0000) 2019-10-18T02:09:02.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:02.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:02.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:02.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:02.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:09:02.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:02.138291+0000) 2019-10-18T02:09:03.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:03.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:03.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:03.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:03.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:09:03.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:03.183036+0000) 2019-10-18T02:09:04.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:04.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:04.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:04.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:05.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:05.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:05.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:05.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:05.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:05.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:06.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:06.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:06.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:06.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:07.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:07.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:07.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:07.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:07.773 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:09:07.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:07.771960+0000) 2019-10-18T02:09:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:08.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:08.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:08.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:08.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:08.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:08.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:08.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:09.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:09.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:09.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:09.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:10.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:10.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:10.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:10.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:10.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:10.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:11.766 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:09:11.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:11.764597+0000) 2019-10-18T02:09:11.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:11.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:11.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:11.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:12.139 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:09:12.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:12.138400+0000) 2019-10-18T02:09:12.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:12.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:12.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:12.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:13.184 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:09:13.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:13.183134+0000) 2019-10-18T02:09:13.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:13.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:13.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:13.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:14.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:14.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:14.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:14.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:15.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:15.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:15.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:15.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:15.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:15.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:16.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:16.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:16.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:16.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:17.774 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:09:17.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:17.772140+0000) 2019-10-18T02:09:17.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:17.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:17.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:17.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:18.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:18.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:18.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:18.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:19.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:19.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:19.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:19.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:20.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:20.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:20.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:20.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:20.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:20.919+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:21.766 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:09:21.761+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:21.764777+0000) 2019-10-18T02:09:21.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:21.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:21.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:21.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:22.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:09:22.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:22.138507+0000) 2019-10-18T02:09:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:22.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:22.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:23.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:09:23.179+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:23.183250+0000) 2019-10-18T02:09:23.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:23.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:23.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:23.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:24.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:24.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:24.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:24.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:25.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:25.919+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:25.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:25.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:25.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:25.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:26.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:26.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:26.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:26.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:27.774 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:09:27.769+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:27.772331+0000) 2019-10-18T02:09:27.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:27.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:27.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:27.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:28.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:28.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:28.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:28.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:29.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:29.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:29.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:29.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:30.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:30.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:30.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:30.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:30.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:30.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:31.767 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:09:31.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:31.764961+0000) 2019-10-18T02:09:31.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:31.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:31.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:31.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:32.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:09:32.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:32.138610+0000) 2019-10-18T02:09:32.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:32.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:32.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:32.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:33.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:09:33.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:33.183349+0000) 2019-10-18T02:09:33.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:33.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:33.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:33.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:34.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:34.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:34.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:34.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:35.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:35.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:35.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:35.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:35.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:35.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:36.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:36.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:36.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:36.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:37.774 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:09:37.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:37.772513+0000) 2019-10-18T02:09:37.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:37.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:37.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:37.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:38.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:38.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:38.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:38.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:39.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:39.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:39.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:39.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:40.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:40.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:40.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:40.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:40.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:40.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:41.766 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:09:41.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:41.765141+0000) 2019-10-18T02:09:41.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:41.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:41.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:41.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:42.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:09:42.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:42.138716+0000) 2019-10-18T02:09:42.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:42.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:42.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:42.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:43.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:09:43.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:43.183447+0000) 2019-10-18T02:09:43.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:43.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:43.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:43.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:44.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:44.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:44.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:44.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:45.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:45.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:45.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:45.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:45.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:45.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:46.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:46.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:46.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:46.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:47.774 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:09:47.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:47.772724+0000) 2019-10-18T02:09:47.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:47.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:47.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:47.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:48.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:48.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:48.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:48.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:49.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:49.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:49.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:49.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:50.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:50.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:50.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:50.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:50.929 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:50.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:51.767 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:09:51.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:51.765334+0000) 2019-10-18T02:09:51.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:51.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:51.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:51.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:52.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:09:52.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:52.138824+0000) 2019-10-18T02:09:52.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:52.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:52.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:52.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:53.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:09:53.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:53.183541+0000) 2019-10-18T02:09:53.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:53.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:53.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:53.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:54.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:54.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:54.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:54.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:55.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:09:55.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:09:55.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:55.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:55.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:55.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:56.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:56.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:56.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:56.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:57.774 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:09:57.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:09:57.772908+0000) 2019-10-18T02:09:57.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:57.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:57.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:57.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:59.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:59.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:59.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:59.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:09:59.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:59.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:09:59.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:09:59.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:00.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:00.926+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:01.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:01.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:01.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:01.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:01.767 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:10:01.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:01.765520+0000) 2019-10-18T02:10:02.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:02.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:02.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:02.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:02.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:10:02.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:02.138931+0000) 2019-10-18T02:10:03.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:03.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:03.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:03.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:03.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:03.178+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:03.183640+0000) 2019-10-18T02:10:04.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:04.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:04.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:04.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:04.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:04.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:04.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:04.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:05.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:05.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:06.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:06.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:06.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:06.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:07.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:07.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:07.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:07.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:07.774 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:10:07.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:07.773085+0000) 2019-10-18T02:10:07.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:07.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:08.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:07.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:08.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:08.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:08.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:08.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:10.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:09.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:10.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:09.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:10.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:10.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:11.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:11.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:11.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:11.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:11.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:10:11.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:11.765704+0000) 2019-10-18T02:10:11.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:11.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:11.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:11.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:12.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:10:12.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:12.139041+0000) 2019-10-18T02:10:13.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:13.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:13.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:13.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:13.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:13.177+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:13.183740+0000) 2019-10-18T02:10:14.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:14.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:14.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:14.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:15.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:15.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:15.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:15.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:16.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:16.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:16.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:16.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:17.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:17.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:17.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:17.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:17.775 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:10:17.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:17.773268+0000) 2019-10-18T02:10:17.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:17.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:17.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:17.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:18.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:18.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:18.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:18.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:19.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:19.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:19.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:19.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:20.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:20.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:20.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:20.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:20.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:20.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:21.767 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:10:21.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:21.765904+0000) 2019-10-18T02:10:21.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:21.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:21.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:21.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:22.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:10:22.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:22.139144+0000) 2019-10-18T02:10:22.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:22.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:22.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:22.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:23.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:23.181+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:23.183835+0000) 2019-10-18T02:10:23.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:23.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:23.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:23.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:24.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:24.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:24.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:24.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:25.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:25.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:25.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:25.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:25.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:25.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:26.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:26.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:26.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:26.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:27.775 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:10:27.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:27.773448+0000) 2019-10-18T02:10:27.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:27.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:27.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:27.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:28.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:28.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:28.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:28.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:29.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:29.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:29.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:29.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:30.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:30.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:30.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:30.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:30.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:30.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:31.767 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:10:31.760+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:31.766088+0000) 2019-10-18T02:10:31.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:31.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:31.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:31.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:32.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:10:32.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:32.139253+0000) 2019-10-18T02:10:32.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:32.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:32.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:32.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:33.185 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:33.181+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:33.183926+0000) 2019-10-18T02:10:33.186 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:33.181+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:33.185026+0000) 2019-10-18T02:10:33.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:33.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:33.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:33.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:34.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:34.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:34.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:34.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:35.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:35.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:35.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:35.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:35.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:35.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:36.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:36.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:36.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:36.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:37.685 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:37.681+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:37.684122+0000) 2019-10-18T02:10:37.775 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:10:37.768+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:37.773630+0000) 2019-10-18T02:10:37.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:37.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:37.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:37.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:38.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:38.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:38.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:38.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:39.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:39.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:39.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:39.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:40.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:40.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:40.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:40.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:40.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:40.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:41.767 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:10:41.764+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:41.766240+0000) 2019-10-18T02:10:41.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:41.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:41.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:41.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:42.140 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:10:42.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:42.139362+0000) 2019-10-18T02:10:42.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:42.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:42.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:42.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:43.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:43.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:43.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:43.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:44.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:44.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:44.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:44.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:45.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:45.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:45.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:45.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:45.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:45.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:46.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:46.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:46.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:46.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:47.685 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:47.680+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:47.684225+0000) 2019-10-18T02:10:47.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:47.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:47.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:47.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:47.775 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:10:47.767+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:47.773814+0000) 2019-10-18T02:10:48.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:48.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:48.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:48.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:49.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:49.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:49.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:49.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:50.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:50.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:50.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:50.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:50.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:50.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:51.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:10:51.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:51.766413+0000) 2019-10-18T02:10:51.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:51.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:51.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:51.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:52.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:10:52.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:52.139469+0000) 2019-10-18T02:10:52.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:52.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:52.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:52.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:53.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:53.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:53.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:53.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:54.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:54.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:54.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:54.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:55.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:55.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:55.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:55.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:55.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:10:55.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:10:56.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:56.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:56.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:56.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:57.685 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:10:57.680+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:57.684332+0000) 2019-10-18T02:10:57.775 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:10:57.767+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:10:57.773993+0000) 2019-10-18T02:10:57.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:57.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:57.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:57.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:58.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:58.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:58.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:58.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:10:59.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:59.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:10:59.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:10:59.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:00.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:00.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:00.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:00.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:00.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:00.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:01.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:11:01.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:01.766562+0000) 2019-10-18T02:11:01.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:01.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:01.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:01.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:02.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:11:02.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:02.139575+0000) 2019-10-18T02:11:02.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:02.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:02.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:02.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:03.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:03.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:03.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:03.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:04.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:04.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:04.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:04.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:05.942 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:05.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:05.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:05.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:05.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:05.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:06.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:06.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:06.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:06.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:07.685 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:11:07.680+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:07.684437+0000) 2019-10-18T02:11:07.775 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:11:07.771+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:07.774174+0000) 2019-10-18T02:11:07.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:07.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:07.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:07.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:08.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:08.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:08.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:08.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:09.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:09.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:09.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:09.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:10.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:10.940+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:11.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:11.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:11.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:11.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:11.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:11:11.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:11.766726+0000) 2019-10-18T02:11:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:12.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:12.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:12.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:11:12.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:12.139681+0000) 2019-10-18T02:11:13.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:13.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:13.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:13.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:14.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:14.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:14.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:14.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:15.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:15.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:15.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:15.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:15.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:15.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:16.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:16.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:16.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:16.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:17.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:17.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:17.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:17.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:17.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:11:17.679+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:17.684543+0000) 2019-10-18T02:11:17.776 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:11:17.771+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:17.774373+0000) 2019-10-18T02:11:18.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:18.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:18.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:18.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:19.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:19.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:19.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:19.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:20.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:20.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:20.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:20.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:20.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:20.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:21.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:21.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:21.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:21.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:21.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:11:21.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:21.766919+0000) 2019-10-18T02:11:22.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:11:22.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:22.139789+0000) 2019-10-18T02:11:22.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:22.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:22.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:22.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:23.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:23.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:23.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:23.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:24.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:24.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:24.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:24.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:25.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:25.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:25.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:25.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:25.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:25.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:26.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:26.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:26.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:26.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:27.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:27.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:27.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:27.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:27.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:11:27.679+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:27.684648+0000) 2019-10-18T02:11:27.776 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:11:27.771+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:27.774524+0000) 2019-10-18T02:11:28.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:28.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:28.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:28.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:29.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:29.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:29.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:29.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:30.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:30.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:30.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:30.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:30.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:30.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:31.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:31.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:31.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:31.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:31.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:11:31.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:31.767099+0000) 2019-10-18T02:11:32.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:11:32.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:32.139896+0000) 2019-10-18T02:11:32.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:32.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:32.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:32.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:33.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:33.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:33.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:33.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:34.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:34.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:34.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:34.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:35.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:35.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:35.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:35.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:35.947 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:35.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:36.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:36.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:36.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:36.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:37.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:37.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:37.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:37.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:37.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:11:37.679+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:37.684754+0000) 2019-10-18T02:11:37.776 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:11:37.771+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:37.774736+0000) 2019-10-18T02:11:38.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:38.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:38.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:38.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:39.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:39.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:39.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:39.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:40.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:40.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:40.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:40.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:40.948 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:40.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:41.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:41.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:41.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:41.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:41.768 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:11:41.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:41.767279+0000) 2019-10-18T02:11:42.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:11:42.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:42.140006+0000) 2019-10-18T02:11:42.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:42.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:42.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:42.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:43.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:43.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:43.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:43.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:44.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:44.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:44.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:44.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:45.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:45.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:45.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:45.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:45.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:45.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:46.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:46.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:46.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:46.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:47.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:47.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:47.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:47.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:47.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:11:47.679+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:47.684860+0000) 2019-10-18T02:11:47.776 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:11:47.771+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:47.774918+0000) 2019-10-18T02:11:48.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:48.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:48.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:48.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:49.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:49.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:49.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:49.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:50.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:50.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:50.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:50.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:50.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:50.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:51.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:51.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:51.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:51.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:51.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:11:51.763+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:51.767459+0000) 2019-10-18T02:11:52.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:11:52.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:52.140117+0000) 2019-10-18T02:11:52.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:52.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:52.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:52.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:53.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:53.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:53.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:53.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:54.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:54.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:54.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:54.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:55.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:55.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:55.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:55.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:55.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:11:55.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:11:56.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:56.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:56.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:56.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:57.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:57.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:57.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:57.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:57.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:11:57.678+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:57.684965+0000) 2019-10-18T02:11:57.776 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:11:57.771+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:11:57.775097+0000) 2019-10-18T02:11:58.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:58.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:58.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:58.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:11:59.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:59.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:11:59.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:11:59.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:00.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:00.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:00.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:00.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:00.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:00.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:01.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:01.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:01.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:01.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:01.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:12:01.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:01.767640+0000) 2019-10-18T02:12:02.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:12:02.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:02.140227+0000) 2019-10-18T02:12:02.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:02.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:02.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:02.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:03.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:03.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:03.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:03.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:04.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:04.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:04.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:04.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:05.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:05.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:05.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:05.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:05.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:05.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:06.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:06.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:06.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:06.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:07.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:07.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:07.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:07.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:07.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:12:07.678+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:07.685067+0000) 2019-10-18T02:12:07.777 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:12:07.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:07.775281+0000) 2019-10-18T02:12:08.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:08.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:08.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:08.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:09.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:09.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:09.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:09.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:10.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:10.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:10.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:10.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:10.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:10.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:11.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:11.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:11.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:11.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:11.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:12:11.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:11.767818+0000) 2019-10-18T02:12:12.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:12:12.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:12.140335+0000) 2019-10-18T02:12:12.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:12.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:12.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:12.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:13.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:13.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:13.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:13.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:14.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:14.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:14.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:14.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:15.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:15.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:15.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:15.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:15.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:15.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:16.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:16.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:16.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:16.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:17.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:17.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:17.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:17.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:17.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:12:17.678+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:17.685171+0000) 2019-10-18T02:12:17.777 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:12:17.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:17.775491+0000) 2019-10-18T02:12:18.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:18.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:18.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:18.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:19.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:19.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:19.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:19.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:20.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:20.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:20.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:20.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:20.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:20.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:21.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:21.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:21.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:21.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:21.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:12:21.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:21.768011+0000) 2019-10-18T02:12:22.141 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:12:22.134+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:22.140447+0000) 2019-10-18T02:12:22.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:22.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:22.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:22.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:23.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:23.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:23.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:23.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:24.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:24.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:24.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:24.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:25.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:25.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:25.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:25.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:25.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:25.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:26.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:26.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:26.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:26.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:27.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:27.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:27.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:27.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:27.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:12:27.677+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:27.685275+0000) 2019-10-18T02:12:27.777 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:12:27.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:27.775710+0000) 2019-10-18T02:12:28.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:28.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:28.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:28.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:29.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:29.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:29.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:29.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:30.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:30.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:30.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:30.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:30.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:30.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:31.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:31.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:31.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:31.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:31.769 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:12:31.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:31.768199+0000) 2019-10-18T02:12:32.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:12:32.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:32.140555+0000) 2019-10-18T02:12:32.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:32.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:32.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:32.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:33.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:33.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:33.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:33.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:34.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:34.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:34.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:34.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:35.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:35.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:35.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:35.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:35.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:35.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:36.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:36.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:36.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:37.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:37.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:37.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:37.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:37.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:12:37.677+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:37.685380+0000) 2019-10-18T02:12:37.777 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:12:37.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:37.775890+0000) 2019-10-18T02:12:38.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:38.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:38.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:38.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:39.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:39.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:39.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:39.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:40.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:40.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:40.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:40.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:40.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:40.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:41.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:41.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:41.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:41.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:41.770 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:12:41.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:41.768400+0000) 2019-10-18T02:12:42.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:12:42.133+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:42.140664+0000) 2019-10-18T02:12:42.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:42.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:42.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:42.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:43.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:43.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:43.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:43.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:44.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:44.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:44.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:44.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:45.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:45.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:45.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:45.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:45.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:45.953+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:46.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:46.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:46.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:46.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:47.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:47.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:47.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:47.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:47.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:12:47.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:47.685481+0000) 2019-10-18T02:12:47.777 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:12:47.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:47.776080+0000) 2019-10-18T02:12:48.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:48.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:48.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:48.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:49.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:49.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:49.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:49.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:50.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:50.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:50.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:50.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:50.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:50.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:51.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:51.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:51.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:51.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:51.770 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:12:51.762+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:51.768584+0000) 2019-10-18T02:12:52.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:12:52.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:52.140778+0000) 2019-10-18T02:12:52.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:52.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:52.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:52.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:53.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:53.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:53.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:53.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:54.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:54.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:54.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:54.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:55.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:55.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:55.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:55.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:55.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:12:55.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:12:56.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:56.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:56.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:56.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:57.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:57.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:57.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:57.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:57.686 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:12:57.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:57.685586+0000) 2019-10-18T02:12:57.778 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:12:57.770+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:12:57.776272+0000) 2019-10-18T02:12:58.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:58.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:58.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:58.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:12:59.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:59.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:12:59.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:12:59.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:00.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:00.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:00.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:00.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:00.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:00.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:01.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:01.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:01.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:01.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:01.770 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:13:01.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:01.768767+0000) 2019-10-18T02:13:02.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:13:02.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:02.140887+0000) 2019-10-18T02:13:02.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:02.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:02.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:02.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:03.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:03.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:03.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:03.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:04.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:04.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:04.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:04.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:05.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:05.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:05.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:05.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:05.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:05.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:06.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:06.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:06.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:06.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:07.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:07.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:07.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:07.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:07.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:13:07.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:07.685689+0000) 2019-10-18T02:13:07.778 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:13:07.774+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:07.776475+0000) 2019-10-18T02:13:08.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:08.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:08.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:08.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:09.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:09.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:09.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:09.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:10.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:10.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:10.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:10.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:10.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:10.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:11.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:11.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:11.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:11.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:11.770 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:13:11.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:11.768916+0000) 2019-10-18T02:13:12.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:13:12.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:12.140993+0000) 2019-10-18T02:13:12.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:12.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:12.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:12.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:13.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:13.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:13.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:13.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:14.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:14.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:14.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:14.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:15.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:15.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:15.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:15.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:15.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:15.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:16.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:16.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:16.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:16.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:17.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:17.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:17.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:13:17.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:17.685791+0000) 2019-10-18T02:13:17.778 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:13:17.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:17.776668+0000) 2019-10-18T02:13:18.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:18.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:18.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:18.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:19.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:19.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:19.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:19.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:20.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:20.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:20.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:20.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:20.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:20.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:21.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:21.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:21.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:21.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:21.774 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:13:21.765+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:21.769133+0000) 2019-10-18T02:13:22.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:13:22.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:22.141104+0000) 2019-10-18T02:13:22.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:22.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:22.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:22.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:23.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:23.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:23.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:23.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:24.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:24.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:24.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:24.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:25.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:25.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:25.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:25.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:25.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:25.964+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:26.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:26.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:26.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:26.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:27.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:27.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:27.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:13:27.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:27.685899+0000) 2019-10-18T02:13:27.779 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:13:27.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:27.776855+0000) 2019-10-18T02:13:28.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:28.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:28.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:28.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:29.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:29.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:29.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:29.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:30.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:30.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:30.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:30.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:30.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:30.964+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:31.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:31.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:31.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:31.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:31.770 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:13:31.765+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:31.769350+0000) 2019-10-18T02:13:32.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:13:32.136+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:32.141205+0000) 2019-10-18T02:13:32.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:32.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:32.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:32.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:33.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:33.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:33.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:33.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:34.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:34.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:34.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:34.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:35.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:35.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:35.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:35.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:35.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:35.964+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:36.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:36.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:36.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:36.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:37.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:37.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:37.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:37.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:37.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:13:37.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:37.686003+0000) 2019-10-18T02:13:37.778 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:13:37.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:37.777041+0000) 2019-10-18T02:13:38.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:38.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:38.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:38.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:39.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:39.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:39.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:39.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:40.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:40.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:40.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:40.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:41.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:41.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:41.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:41.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:41.771 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:13:41.765+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:41.769556+0000) 2019-10-18T02:13:42.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:13:42.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:42.141310+0000) 2019-10-18T02:13:42.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:42.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:42.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:42.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:43.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:43.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:43.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:43.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:44.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:44.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:44.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:44.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:45.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:45.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:45.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:45.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:45.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:45.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:46.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:46.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:46.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:46.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:47.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:47.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:47.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:47.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:47.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:13:47.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:47.686105+0000) 2019-10-18T02:13:47.778 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:13:47.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:47.777276+0000) 2019-10-18T02:13:48.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:48.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:48.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:48.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:49.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:49.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:49.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:49.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:50.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:50.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:50.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:50.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:50.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:50.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:51.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:51.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:51.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:51.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:51.771 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:13:51.765+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:51.769746+0000) 2019-10-18T02:13:52.142 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:13:52.135+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:52.141419+0000) 2019-10-18T02:13:52.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:52.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:52.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:52.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:53.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:53.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:53.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:53.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:54.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:54.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:54.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:54.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:55.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:55.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:55.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:55.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:55.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:13:55.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:13:56.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:56.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:56.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:56.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:57.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:57.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:57.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:57.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:57.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:13:57.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:57.686207+0000) 2019-10-18T02:13:57.779 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:13:57.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:13:57.777465+0000) 2019-10-18T02:13:58.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:58.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:58.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:58.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:13:59.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:59.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:13:59.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:13:59.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:00.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:00.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:00.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:00.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:00.972 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:00.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:01.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:01.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:01.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:01.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:01.771 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:14:01.765+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:01.769938+0000) 2019-10-18T02:14:02.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:14:02.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:02.141622+0000) 2019-10-18T02:14:02.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:02.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:02.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:02.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:03.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:03.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:03.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:03.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:04.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:04.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:04.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:04.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:05.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:05.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:05.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:05.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:05.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:05.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:06.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:06.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:06.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:06.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:07.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:07.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:07.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:07.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:07.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:14:07.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:07.686309+0000) 2019-10-18T02:14:07.780 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:14:07.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:07.777648+0000) 2019-10-18T02:14:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:08.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:08.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:09.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:09.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:09.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:09.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:10.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:10.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:10.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:10.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:10.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:10.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:11.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:11.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:11.771 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:14:11.765+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:11.770132+0000) 2019-10-18T02:14:12.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:14:12.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:12.141734+0000) 2019-10-18T02:14:12.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:12.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:12.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:12.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:13.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:13.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:13.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:13.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:14.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:14.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:14.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:14.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:15.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:15.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:15.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:15.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:15.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:15.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:16.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:16.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:16.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:16.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:17.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:17.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:17.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:17.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:17.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:14:17.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:17.686414+0000) 2019-10-18T02:14:17.779 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:14:17.773+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:17.777828+0000) 2019-10-18T02:14:18.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:18.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:18.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:18.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:19.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:19.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:19.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:20.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:20.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:20.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:20.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:20.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:20.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:21.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:21.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:21.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:21.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:21.771 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:14:21.769+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:21.770308+0000) 2019-10-18T02:14:22.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:14:22.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:22.141838+0000) 2019-10-18T02:14:22.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:22.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:22.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:22.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:23.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:23.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:23.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:23.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:24.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:24.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:24.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:24.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:25.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:25.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:25.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:25.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:25.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:25.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:26.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:26.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:26.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:26.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:27.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:27.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:27.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:27.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:27.687 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:14:27.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:27.686552+0000) 2019-10-18T02:14:27.779 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:14:27.772+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:27.778038+0000) 2019-10-18T02:14:28.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:28.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:28.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:28.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:29.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:29.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:29.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:29.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:30.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:30.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:30.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:30.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:30.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:30.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:31.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:31.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:31.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:31.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:31.771 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:14:31.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:31.770460+0000) 2019-10-18T02:14:32.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:14:32.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:32.141942+0000) 2019-10-18T02:14:32.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:32.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:32.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:32.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:33.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:33.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:33.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:33.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:34.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:34.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:34.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:34.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:35.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:35.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:35.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:35.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:35.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:35.974+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:36.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:36.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:36.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:36.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:37.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:37.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:37.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:37.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:37.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:14:37.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:37.686655+0000) 2019-10-18T02:14:37.779 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:14:37.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:37.778224+0000) 2019-10-18T02:14:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:38.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:38.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:39.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:39.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:39.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:39.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:40.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:40.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:40.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:40.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:40.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:40.974+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:41.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:41.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:41.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:41.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:41.772 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:14:41.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:41.770607+0000) 2019-10-18T02:14:42.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:42.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:42.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:42.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:42.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:14:42.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:42.142049+0000) 2019-10-18T02:14:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:43.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:43.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:44.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:44.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:44.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:44.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:45.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:45.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:45.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:45.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:45.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:45.974+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:46.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:46.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:46.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:46.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:47.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:47.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:47.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:47.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:47.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:14:47.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:47.686761+0000) 2019-10-18T02:14:47.779 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:14:47.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:47.778392+0000) 2019-10-18T02:14:48.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:48.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:48.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:48.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:49.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:49.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:49.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:49.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:50.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:50.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:50.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:50.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:50.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:50.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:51.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:51.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:51.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:51.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:51.772 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:14:51.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:51.770819+0000) 2019-10-18T02:14:52.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:14:52.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:52.142157+0000) 2019-10-18T02:14:52.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:52.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:52.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:52.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:53.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:53.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:53.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:53.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:54.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:54.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:54.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:54.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:55.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:55.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:55.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:55.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:55.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:14:55.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:14:56.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:56.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:56.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:56.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:57.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:57.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:57.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:57.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:14:57.681+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:57.686871+0000) 2019-10-18T02:14:57.780 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:14:57.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:14:57.778552+0000) 2019-10-18T02:14:58.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:58.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:58.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:58.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:14:59.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:59.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:14:59.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:14:59.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:00.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:00.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:00.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:00.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:00.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:00.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:01.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:01.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:01.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:01.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:01.772 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:15:01.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:01.771002+0000) 2019-10-18T02:15:02.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:15:02.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:02.142263+0000) 2019-10-18T02:15:02.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:02.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:02.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:02.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:03.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:03.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:03.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:03.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:04.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:04.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:04.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:04.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:05.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:05.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:05.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:05.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:05.984 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:05.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:06.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:06.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:06.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:06.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:07.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:07.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:07.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:07.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:07.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:15:07.681+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:07.686975+0000) 2019-10-18T02:15:07.780 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:07.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:07.778733+0000) 2019-10-18T02:15:08.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:08.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:08.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:09.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:09.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:09.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:09.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:10.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:10.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:10.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:10.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:10.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:10.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:11.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:11.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:11.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:11.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:11.772 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:15:11.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:11.771187+0000) 2019-10-18T02:15:12.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:15:12.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:12.142373+0000) 2019-10-18T02:15:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:12.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:12.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:13.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:13.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:13.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:13.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:14.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:14.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:14.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:14.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:15.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:15.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:15.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:15.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:15.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:15.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:16.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:16.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:16.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:16.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:17.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:17.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:17.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:17.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:17.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:15:17.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:17.687096+0000) 2019-10-18T02:15:17.780 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:17.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:17.778912+0000) 2019-10-18T02:15:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:18.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:18.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:19.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:19.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:19.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:19.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:20.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:20.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:20.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:20.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:20.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:20.985+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:21.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:21.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:21.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:21.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:21.772 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:15:21.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:21.771322+0000) 2019-10-18T02:15:22.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:15:22.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:22.142486+0000) 2019-10-18T02:15:22.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:22.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:22.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:22.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:23.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:23.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:23.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:23.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:24.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:24.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:24.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:24.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:25.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:25.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:25.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:25.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:25.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:25.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:26.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:26.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:26.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:26.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:27.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:27.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:27.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:27.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:27.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:15:27.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:27.687207+0000) 2019-10-18T02:15:27.780 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:27.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:27.779095+0000) 2019-10-18T02:15:28.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:28.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:28.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:28.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:29.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:29.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:29.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:29.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:30.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:30.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:30.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:30.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:30.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:30.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:31.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:31.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:31.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:31.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:31.772 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:15:31.768+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:31.771505+0000) 2019-10-18T02:15:32.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:15:32.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:32.142593+0000) 2019-10-18T02:15:32.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:32.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:32.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:32.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:33.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:33.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:33.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:33.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:34.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:34.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:34.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:34.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:35.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:35.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:35.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:35.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:35.989 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:35.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:36.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:36.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:36.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:36.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:37.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:37.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:37.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:37.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:37.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:15:37.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:37.687308+0000) 2019-10-18T02:15:37.782 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:37.776+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:37.779295+0000) 2019-10-18T02:15:38.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:38.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:38.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:38.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:39.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:39.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:39.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:39.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:40.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:40.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:40.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:40.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:40.990 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:40.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:41.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:41.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:41.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:41.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:41.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:15:41.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:41.771688+0000) 2019-10-18T02:15:42.143 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:15:42.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:42.142696+0000) 2019-10-18T02:15:42.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:42.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:42.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:42.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:43.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:43.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:43.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:43.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:44.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:44.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:44.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:44.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:45.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:45.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:45.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:45.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:45.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:45.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:46.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:46.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:46.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:46.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:47.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:47.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:47.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:47.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:47.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:15:47.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:47.687413+0000) 2019-10-18T02:15:47.780 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:47.775+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:47.779470+0000) 2019-10-18T02:15:47.781 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:47.775+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:47.780454+0000) 2019-10-18T02:15:48.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:48.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:48.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:48.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:49.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:49.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:49.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:49.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:50.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:50.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:50.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:50.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:50.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:50.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:51.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:51.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:51.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:51.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:51.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:15:51.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:51.771871+0000) 2019-10-18T02:15:52.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:15:52.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:52.142803+0000) 2019-10-18T02:15:52.281 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:15:52.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:52.279856+0000) 2019-10-18T02:15:52.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:52.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:52.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:52.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:53.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:53.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:53.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:53.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:54.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:54.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:54.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:54.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:55.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:55.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:55.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:55.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:55.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:15:55.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:15:56.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:56.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:56.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:56.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:57.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:57.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:57.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:57.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:57.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:15:57.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:15:57.687517+0000) 2019-10-18T02:15:58.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:58.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:58.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:58.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:15:59.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:59.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:15:59.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:15:59.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:00.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:00.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:00.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:00.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:00.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:00.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:01.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:01.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:01.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:01.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:01.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:16:01.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:01.772051+0000) 2019-10-18T02:16:02.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:16:02.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:02.142907+0000) 2019-10-18T02:16:02.281 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:16:02.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:02.280068+0000) 2019-10-18T02:16:02.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:02.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:02.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:02.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:03.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:03.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:03.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:03.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:04.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:04.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:04.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:04.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:05.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:05.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:05.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:05.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:05.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:05.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:06.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:06.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:06.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:06.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:07.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:07.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:07.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:07.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:07.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:16:07.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:07.687620+0000) 2019-10-18T02:16:08.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:08.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:08.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:08.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:09.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:09.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:09.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:09.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:10.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:10.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:10.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:10.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:10.995 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:10.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:11.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:11.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:11.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:11.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:11.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:16:11.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:11.772191+0000) 2019-10-18T02:16:12.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:16:12.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:12.143015+0000) 2019-10-18T02:16:12.281 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:16:12.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:12.280277+0000) 2019-10-18T02:16:12.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:12.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:12.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:12.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:13.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:13.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:13.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:13.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:14.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:14.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:14.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:14.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:15.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:15.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:15.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:15.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:15.996 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:15.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:16.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:16.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:16.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:16.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:17.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:17.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:17.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:17.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:17.688 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:16:17.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:17.687724+0000) 2019-10-18T02:16:18.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:18.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:18.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:18.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:19.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:19.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:19.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:19.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:20.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:20.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:20.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:20.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:20.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:20.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:21.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:21.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:21.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:21.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:21.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:16:21.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:21.772372+0000) 2019-10-18T02:16:22.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:16:22.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:22.143123+0000) 2019-10-18T02:16:22.281 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:16:22.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:22.280459+0000) 2019-10-18T02:16:22.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:22.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:22.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:22.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:23.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:23.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:23.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:23.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:24.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:24.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:24.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:24.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:25.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:25.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:25.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:25.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:25.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:25.995+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:26.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:26.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:26.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:26.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:27.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:27.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:27.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:27.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:27.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:16:27.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:27.687832+0000) 2019-10-18T02:16:28.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:28.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:28.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:28.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:29.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:29.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:29.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:29.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:30.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:30.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:30.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:30.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:30.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:30.995+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:31.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:31.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:31.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:31.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:31.773 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:16:31.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:31.772562+0000) 2019-10-18T02:16:32.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:16:32.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:32.143230+0000) 2019-10-18T02:16:32.282 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:16:32.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:32.280643+0000) 2019-10-18T02:16:32.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:32.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:32.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:32.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:33.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:33.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:33.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:33.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:34.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:34.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:34.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:34.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:35.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:35.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:35.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:35.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:36.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:35.995+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:36.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:36.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:36.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:36.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:37.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:16:37.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:37.687935+0000) 2019-10-18T02:16:37.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:37.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:37.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:37.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:38.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:38.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:38.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:38.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:39.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:39.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:40.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:40.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:40.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:40.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:41.001 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:40.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:41.774 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:16:41.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:41.772745+0000) 2019-10-18T02:16:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:41.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:41.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:42.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:16:42.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:42.143334+0000) 2019-10-18T02:16:42.282 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:16:42.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:42.280823+0000) 2019-10-18T02:16:42.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:42.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:42.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:42.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:43.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:43.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:43.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:43.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:44.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:44.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:44.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:44.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:45.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:45.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:45.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:45.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:46.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:45.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:46.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:46.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:46.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:46.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:47.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:16:47.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:47.688033+0000) 2019-10-18T02:16:47.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:47.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:47.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:47.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:48.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:48.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:48.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:48.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:49.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:49.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:49.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:49.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:50.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:50.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:50.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:50.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:51.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:50.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:51.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:51.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:51.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:51.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:51.774 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:16:51.767+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:51.772927+0000) 2019-10-18T02:16:52.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:16:52.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:52.143440+0000) 2019-10-18T02:16:52.282 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:16:52.275+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:52.281029+0000) 2019-10-18T02:16:52.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:52.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:52.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:52.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:53.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:53.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:53.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:53.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:54.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:54.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:54.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:54.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:55.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:55.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:55.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:55.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:56.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:16:55.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:16:56.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:56.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:56.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:56.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:57.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:16:57.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:16:57.688135+0000) 2019-10-18T02:16:57.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:57.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:57.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:57.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:58.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:58.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:58.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:58.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:16:59.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:59.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:16:59.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:16:59.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:00.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:00.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:00.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:00.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:01.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:00.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:01.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:01.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:01.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:01.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:01.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:17:01.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:01.773111+0000) 2019-10-18T02:17:02.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:17:02.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:02.143546+0000) 2019-10-18T02:17:02.282 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:17:02.274+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:02.281219+0000) 2019-10-18T02:17:02.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:02.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:02.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:02.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:03.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:03.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:03.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:03.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:04.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:04.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:04.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:04.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:05.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:05.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:05.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:05.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:06.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:05.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:06.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:06.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:06.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:06.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:07.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:17:07.682+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:07.688251+0000) 2019-10-18T02:17:07.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:07.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:07.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:07.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:08.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:08.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:08.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:08.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:09.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:09.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:09.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:09.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:10.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:10.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:10.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:10.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:11.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:10.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:11.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:17:11.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:11.773293+0000) 2019-10-18T02:17:11.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:11.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:11.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:11.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:12.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:17:12.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:12.143658+0000) 2019-10-18T02:17:12.283 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:17:12.274+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:12.281401+0000) 2019-10-18T02:17:12.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:12.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:12.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:12.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:13.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:13.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:13.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:13.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:14.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:14.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:14.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:14.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:15.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:15.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:15.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:15.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:16.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:16.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:16.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:16.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:16.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:16.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:17.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:17:17.681+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:17.688365+0000) 2019-10-18T02:17:17.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:17.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:17.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:17.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:18.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:18.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:18.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:18.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:19.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:19.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:19.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:19.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:20.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:20.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:20.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:20.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:21.007 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:21.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:21.774 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:17:21.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:21.773481+0000) 2019-10-18T02:17:21.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:21.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:21.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:21.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:22.144 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:17:22.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:22.143767+0000) 2019-10-18T02:17:22.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:17:22.274+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:22.281609+0000) 2019-10-18T02:17:22.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:22.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:22.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:22.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:23.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:23.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:23.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:23.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:24.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:24.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:24.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:24.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:25.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:25.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:25.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:25.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:26.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:26.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:26.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:26.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:26.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:26.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:27.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:17:27.681+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:27.688474+0000) 2019-10-18T02:17:27.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:27.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:27.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:27.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:28.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:28.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:28.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:28.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:30.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:29.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:30.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:29.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:30.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:30.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:30.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:30.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:31.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:31.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:31.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:17:31.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:31.773662+0000) 2019-10-18T02:17:32.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:32.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:32.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:32.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:32.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:17:32.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:32.143873+0000) 2019-10-18T02:17:32.283 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:17:32.274+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:32.281789+0000) 2019-10-18T02:17:33.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:33.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:33.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:34.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:34.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:34.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:34.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:35.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:35.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:35.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:35.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:36.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:36.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:36.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:36.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:36.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:36.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:37.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:37.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:37.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:37.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:37.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:17:37.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:37.688580+0000) 2019-10-18T02:17:38.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:38.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:38.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:38.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:39.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:39.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:39.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:39.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:40.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:40.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:40.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:40.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:41.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:41.009+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:41.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:41.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:41.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:17:41.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:41.773846+0000) 2019-10-18T02:17:42.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:42.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:42.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:42.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:42.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:17:42.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:42.143979+0000) 2019-10-18T02:17:42.283 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:17:42.274+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:42.281973+0000) 2019-10-18T02:17:43.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:43.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:43.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:43.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:44.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:44.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:44.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:44.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:45.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:45.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:45.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:45.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:46.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:46.009+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:46.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:46.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:46.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:46.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:47.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:47.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:47.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:47.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:47.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:17:47.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:47.688686+0000) 2019-10-18T02:17:48.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:48.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:48.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:48.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:49.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:49.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:49.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:49.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:50.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:50.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:50.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:50.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:51.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:51.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:51.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:51.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:51.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:51.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:51.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:17:51.766+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:51.774026+0000) 2019-10-18T02:17:52.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:17:52.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:52.144095+0000) 2019-10-18T02:17:52.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:52.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:52.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:52.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:52.283 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:17:52.274+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:52.282154+0000) 2019-10-18T02:17:53.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:53.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:53.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:53.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:54.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:54.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:54.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:54.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:55.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:55.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:56.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:17:56.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:17:56.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:56.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:56.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:56.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:57.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:57.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:57.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:57.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:57.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:17:57.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:17:57.688793+0000) 2019-10-18T02:17:58.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:58.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:58.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:58.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:17:59.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:59.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:17:59.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:17:59.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:00.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:00.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:00.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:00.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:01.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:01.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:01.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:01.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:01.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:01.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:01.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:18:01.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:01.774207+0000) 2019-10-18T02:18:02.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:18:02.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:02.144208+0000) 2019-10-18T02:18:02.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:02.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:02.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:02.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:02.283 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:18:02.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:02.282337+0000) 2019-10-18T02:18:03.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:03.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:03.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:03.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:04.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:04.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:04.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:04.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:05.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:05.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:05.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:05.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:06.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:06.012+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:06.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:06.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:06.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:06.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:07.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:07.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:07.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:07.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:07.689 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:18:07.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:07.688899+0000) 2019-10-18T02:18:08.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:08.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:08.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:08.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:09.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:09.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:09.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:09.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:10.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:10.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:10.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:10.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:11.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:11.012+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:11.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:11.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:11.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:11.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:11.775 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:18:11.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:11.774367+0000) 2019-10-18T02:18:12.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:18:12.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:12.144323+0000) 2019-10-18T02:18:12.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:12.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:12.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:12.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:12.283 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:18:12.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:12.282495+0000) 2019-10-18T02:18:13.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:13.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:13.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:13.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:14.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:14.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:14.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:14.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:15.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:15.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:15.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:15.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:16.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:16.012+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:16.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:16.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:16.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:16.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:17.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:17.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:17.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:17.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:17.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:18:17.684+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:17.689004+0000) 2019-10-18T02:18:18.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:18.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:07:40.841800+0000 (oldest deadline 2019-10-18T02:08:00.841800+0000) 2019-10-18T02:18:18.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:18.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:19.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:19.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:20.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:20.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:21.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:21.012+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:21.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:21.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:21.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:18:21.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:21.774547+0000) 2019-10-18T02:18:22.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:22.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:22.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:18:22.140+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:22.144426+0000) 2019-10-18T02:18:22.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:18:22.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:22.282674+0000) 2019-10-18T02:18:23.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:23.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:24.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:25.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:25.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:26.019 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:26.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:26.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:26.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:27.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:27.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:27.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:18:27.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:27.689107+0000) 2019-10-18T02:18:28.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:28.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:29.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:29.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:30.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:30.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:31.019 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:31.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:31.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:31.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:31.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:18:31.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:31.774710+0000) 2019-10-18T02:18:32.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:18:32.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:32.144526+0000) 2019-10-18T02:18:32.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:32.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:32.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:18:32.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:32.282854+0000) 2019-10-18T02:18:33.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:33.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:34.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:34.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:35.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:35.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:36.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:36.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:36.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:36.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:37.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:37.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:37.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:18:37.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:37.689211+0000) 2019-10-18T02:18:38.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:38.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:39.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:39.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:40.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:40.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:40.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:40.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:41.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:41.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:41.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:41.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:41.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:41.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:41.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:18:41.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:41.774887+0000) 2019-10-18T02:18:42.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:18:42.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:42.144637+0000) 2019-10-18T02:18:42.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:42.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:42.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:42.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:42.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:18:42.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:42.283040+0000) 2019-10-18T02:18:43.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:43.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:43.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:43.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:44.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:44.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:44.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:44.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:45.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:45.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:45.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:45.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:46.022 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:46.019+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:46.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:46.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:46.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:46.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:47.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:47.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:47.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:47.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:47.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:18:47.683+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:47.689317+0000) 2019-10-18T02:18:48.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:48.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:48.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:48.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:49.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:49.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:49.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:49.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:50.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:50.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:50.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:50.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 288 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:51.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:51.019+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:51.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:51.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:51.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:51.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:51.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:18:51.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:51.775072+0000) 2019-10-18T02:18:52.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:18:52.139+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:52.144742+0000) 2019-10-18T02:18:52.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:52.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:52.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:52.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:52.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:18:52.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:52.283279+0000) 2019-10-18T02:18:53.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:53.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:53.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:53.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:54.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:54.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:54.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:54.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:55.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:55.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:56.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:18:56.019+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:18:56.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:56.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:56.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:56.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:57.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:57.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:57.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:57.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:57.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:18:57.687+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:18:57.689490+0000) 2019-10-18T02:18:58.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:58.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:58.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:58.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:18:59.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:59.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:18:59.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:18:59.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:00.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:00.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:00.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:00.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:01.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:01.018+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:01.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:01.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:01.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:01.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:01.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:19:01.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:01.775287+0000) 2019-10-18T02:19:02.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:19:02.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:02.144840+0000) 2019-10-18T02:19:02.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:19:02.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:02.283463+0000) 2019-10-18T02:19:02.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:02.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:02.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:02.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:03.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:03.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:03.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:03.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:04.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:04.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:04.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:04.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:05.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:05.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:05.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:05.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:06.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:06.018+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:06.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:06.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:06.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:06.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:07.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:07.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:07.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:07.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:07.690 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:19:07.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:07.689601+0000) 2019-10-18T02:19:08.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:08.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:08.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:08.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:09.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:09.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:09.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:09.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:10.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:10.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:10.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:10.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:11.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:11.022+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:11.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:11.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:11.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:11.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:11.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:19:11.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:11.775494+0000) 2019-10-18T02:19:12.145 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:19:12.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:12.144949+0000) 2019-10-18T02:19:12.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:19:12.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:12.283657+0000) 2019-10-18T02:19:12.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:12.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:12.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:12.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:13.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:13.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:13.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:13.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:14.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:14.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:14.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:14.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:15.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:15.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:15.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:15.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:16.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:16.022+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:16.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:16.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:16.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:16.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:17.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:17.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:17.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:17.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:17.691 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:19:17.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:17.690766+0000) 2019-10-18T02:19:18.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:18.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:18.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:18.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:19.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:19.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:19.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:19.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:20.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:20.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:20.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:20.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:21.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:21.022+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:21.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:21.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:21.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:21.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:21.776 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:19:21.773+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:21.775676+0000) 2019-10-18T02:19:22.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:19:22.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:22.145055+0000) 2019-10-18T02:19:22.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:22.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:22.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:22.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:22.284 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:19:22.281+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:22.283842+0000) 2019-10-18T02:19:23.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:23.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:23.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:23.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:24.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:24.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:24.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:24.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:25.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:25.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:25.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:25.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:26.029 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:26.022+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:26.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:26.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:26.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:26.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:27.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:27.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:27.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:27.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:27.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:19:27.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:27.690867+0000) 2019-10-18T02:19:28.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:28.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:28.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:28.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:29.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:29.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:29.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:29.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:30.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:30.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:30.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:30.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:31.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:31.022+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:31.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:31.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:31.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:31.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:31.777 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:19:31.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:31.775836+0000) 2019-10-18T02:19:32.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:19:32.138+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:32.145159+0000) 2019-10-18T02:19:32.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:32.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:32.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:32.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:32.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:19:32.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:32.284024+0000) 2019-10-18T02:19:33.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:33.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:33.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:33.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:34.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:34.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:34.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:34.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:35.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:35.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:35.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:35.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:36.031 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:36.026+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:36.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:36.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:36.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:36.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:37.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:37.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:37.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:37.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:37.691 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:19:37.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:37.690969+0000) 2019-10-18T02:19:38.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:38.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:38.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:38.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:39.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:39.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:39.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:39.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:40.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:40.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:40.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:40.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:41.031 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:41.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:41.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:41.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:41.777 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:19:41.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:41.776049+0000) 2019-10-18T02:19:42.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:19:42.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:42.145269+0000) 2019-10-18T02:19:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:42.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:42.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:42.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:42.285 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:19:42.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:42.284203+0000) 2019-10-18T02:19:43.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:43.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:43.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:43.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:44.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:44.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:44.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:44.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:45.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:45.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:45.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:45.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:46.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:46.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:46.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:46.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:46.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:46.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:47.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:47.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:47.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:47.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:47.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:19:47.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:47.691078+0000) 2019-10-18T02:19:48.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:48.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:48.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:48.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:49.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:49.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:49.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:49.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:50.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:50.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:50.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:50.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:51.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:51.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:51.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:51.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:51.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:51.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:51.777 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:19:51.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:51.776234+0000) 2019-10-18T02:19:52.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:19:52.137+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:52.145377+0000) 2019-10-18T02:19:52.285 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:19:52.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:52.284351+0000) 2019-10-18T02:19:52.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:52.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:52.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:52.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:53.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:53.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:53.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:53.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:54.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:54.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:54.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:54.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:55.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:55.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:55.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:55.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:56.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:19:56.029+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:19:56.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:56.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:56.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:56.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:57.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:57.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:57.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:57.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:57.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:19:57.689+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:19:57.691182+0000) 2019-10-18T02:19:58.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:58.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:58.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:58.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:19:59.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:59.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:19:59.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:19:59.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:00.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:00.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:00.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:00.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:01.035 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:01.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:01.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:01.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:01.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:01.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:01.777 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:20:01.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:01.776463+0000) 2019-10-18T02:20:02.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:20:02.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:02.145488+0000) 2019-10-18T02:20:02.285 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:20:02.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:02.284537+0000) 2019-10-18T02:20:02.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:02.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:02.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:02.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:03.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:03.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:03.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:03.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:04.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:04.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:04.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:04.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:05.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:05.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:05.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:05.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:06.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:06.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:06.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:06.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:06.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:06.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:07.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:07.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:07.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:07.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:07.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:20:07.689+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:07.691290+0000) 2019-10-18T02:20:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:08.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:08.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:09.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:09.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:09.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:09.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:10.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:10.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:10.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:10.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:11.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:11.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:11.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:11.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:11.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:11.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:11.781 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:20:11.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:11.776613+0000) 2019-10-18T02:20:12.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:20:12.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:12.145600+0000) 2019-10-18T02:20:12.285 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:20:12.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:12.284720+0000) 2019-10-18T02:20:12.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:12.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:12.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:12.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:13.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:13.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:13.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:13.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:14.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:14.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:14.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:14.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:15.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:15.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:15.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:15.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:16.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:16.032+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:16.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:16.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:16.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:16.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:17.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:17.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:17.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:20:17.688+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:17.691390+0000) 2019-10-18T02:20:18.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:18.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:18.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:18.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:19.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:19.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:19.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:19.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:20.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:20.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:20.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:20.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:21.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:21.036+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:21.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:21.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:21.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:21.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:21.778 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:20:21.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:21.776812+0000) 2019-10-18T02:20:22.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:20:22.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:22.145706+0000) 2019-10-18T02:20:22.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:20:22.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:22.284898+0000) 2019-10-18T02:20:22.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:22.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:22.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:22.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:23.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:23.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:23.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:23.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:24.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:24.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:24.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:24.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:25.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:25.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:25.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:25.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:26.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:26.036+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:26.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:26.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:26.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:26.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:27.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:27.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:27.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:27.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:27.696 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:20:27.688+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:27.691495+0000) 2019-10-18T02:20:28.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:28.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:28.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:28.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:29.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:29.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:29.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:29.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:30.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:30.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:30.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:30.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:31.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:31.036+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:31.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:31.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:31.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:31.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:31.778 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:20:31.772+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:31.776996+0000) 2019-10-18T02:20:32.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:20:32.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:32.145814+0000) 2019-10-18T02:20:32.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:20:32.280+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:32.285078+0000) 2019-10-18T02:20:32.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:32.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:32.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:32.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:33.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:33.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:33.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:33.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:34.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:34.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:34.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:34.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:35.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:35.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:35.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:35.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:36.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:36.036+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:36.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:36.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:36.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:36.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:37.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:37.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:37.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:37.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:37.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:20:37.688+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:37.691597+0000) 2019-10-18T02:20:38.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:38.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:38.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:38.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:39.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:39.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:39.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:39.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:40.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:40.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:40.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:40.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:41.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:41.036+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:41.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:41.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:41.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:41.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:41.778 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:20:41.771+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:41.777183+0000) 2019-10-18T02:20:42.146 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:20:42.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:42.145921+0000) 2019-10-18T02:20:42.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:20:42.279+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:42.285260+0000) 2019-10-18T02:20:42.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:42.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:42.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:42.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:43.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:43.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:43.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:43.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:44.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:44.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:44.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:44.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:45.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:45.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:45.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:45.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:46.043 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:46.040+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:46.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:46.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:46.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:46.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:47.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:20:47.688+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:47.691699+0000) 2019-10-18T02:20:47.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:47.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:47.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:47.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:48.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:48.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:49.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:49.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:49.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:49.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:50.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:50.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:50.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:50.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:51.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:51.039+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:51.778 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:20:51.771+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:51.777397+0000) 2019-10-18T02:20:51.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:51.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:51.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:51.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:52.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:20:52.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:52.146028+0000) 2019-10-18T02:20:52.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:20:52.279+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:52.285465+0000) 2019-10-18T02:20:52.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:52.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:52.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:52.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:53.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:53.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:53.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:53.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:54.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:54.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:54.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:54.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:55.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:55.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:55.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:55.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:56.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:20:56.039+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:20:56.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:56.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:56.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:56.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:57.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:20:57.687+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:20:57.691806+0000) 2019-10-18T02:20:57.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:57.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:57.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:57.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:58.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:58.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:58.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:58.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:20:59.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:59.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:20:59.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:20:59.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:00.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:00.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:00.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:00.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:01.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:01.039+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:01.778 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:21:01.771+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:01.777619+0000) 2019-10-18T02:21:01.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:01.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:01.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:01.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:02.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:21:02.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:02.146133+0000) 2019-10-18T02:21:02.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:21:02.279+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:02.285647+0000) 2019-10-18T02:21:02.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:02.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:02.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:02.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:03.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:03.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:03.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:03.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:04.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:04.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:04.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:04.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:05.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:05.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:05.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:05.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:06.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:06.043+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:06.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:06.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:06.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:06.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:07.692 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:21:07.687+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:07.691908+0000) 2019-10-18T02:21:07.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:07.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:07.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:08.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:08.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:08.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:08.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:09.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:09.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:09.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:09.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:10.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:10.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:10.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:10.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:11.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:11.043+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:11.778 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:21:11.771+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:11.777822+0000) 2019-10-18T02:21:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:11.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:11.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:12.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:21:12.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:12.146246+0000) 2019-10-18T02:21:12.286 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:21:12.279+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:12.285831+0000) 2019-10-18T02:21:12.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:12.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:12.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:12.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:13.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:13.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:14.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:14.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:14.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:14.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:15.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:15.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:15.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:15.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:16.048 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:16.043+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:16.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:16.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:16.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:16.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:17.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:21:17.687+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:17.692009+0000) 2019-10-18T02:21:17.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:17.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:17.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:17.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:18.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:18.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:18.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:18.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:19.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:19.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:19.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:19.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:20.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:20.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:20.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:20.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:21.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:21.043+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:21.779 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:21:21.771+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:21.778005+0000) 2019-10-18T02:21:21.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:21.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:21.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:21.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:22.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:21:22.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:22.146354+0000) 2019-10-18T02:21:22.287 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:21:22.279+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:22.286008+0000) 2019-10-18T02:21:22.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:22.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:22.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:22.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:23.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:23.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:23.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:23.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:24.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:24.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:24.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:24.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:25.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:25.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:25.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:25.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:26.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:26.042+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:26.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:26.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:26.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:26.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:27.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:21:27.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:27.692115+0000) 2019-10-18T02:21:28.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:28.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:28.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:28.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:29.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:29.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:29.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:29.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:30.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:30.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:30.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:30.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:31.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:31.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:31.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:31.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:31.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:31.046+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:31.779 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:21:31.775+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:31.778216+0000) 2019-10-18T02:21:32.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:32.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:32.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:32.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:32.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:21:32.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:32.146460+0000) 2019-10-18T02:21:32.287 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:21:32.279+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:32.286158+0000) 2019-10-18T02:21:33.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:33.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:33.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:33.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:34.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:34.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:34.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:34.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:35.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:35.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:35.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:35.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:36.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:36.046+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:36.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:36.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:36.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:36.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:37.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:37.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:37.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:37.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:37.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:21:37.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:37.692218+0000) 2019-10-18T02:21:38.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:38.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:38.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:38.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:39.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:39.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:40.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:40.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:40.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:40.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:41.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:41.046+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:41.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:41.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:41.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:41.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:41.779 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:21:41.775+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:41.778369+0000) 2019-10-18T02:21:42.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:21:42.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:42.146568+0000) 2019-10-18T02:21:42.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:42.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:42.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:42.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:42.287 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:21:42.283+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:42.286355+0000) 2019-10-18T02:21:43.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:43.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:43.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:43.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:44.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:44.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:44.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:44.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:45.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:45.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:45.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:45.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:46.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:46.046+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:46.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:46.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:46.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:46.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:47.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:47.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:47.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:47.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:47.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:21:47.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:47.692318+0000) 2019-10-18T02:21:48.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:48.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:48.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:48.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:49.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:49.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:49.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:49.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:50.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:50.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:50.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:50.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:51.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:51.050+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:51.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:51.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:51.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:51.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:51.779 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:21:51.775+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:51.778517+0000) 2019-10-18T02:21:52.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:21:52.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:52.146675+0000) 2019-10-18T02:21:52.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:52.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:52.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:52.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:52.287 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:21:52.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:52.286495+0000) 2019-10-18T02:21:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:53.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:53.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:54.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:54.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:54.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:54.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:55.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:55.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:55.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:55.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:56.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:21:56.050+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:21:56.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:56.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:56.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:56.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:57.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:57.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:57.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:57.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:57.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:21:57.686+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:21:57.692420+0000) 2019-10-18T02:21:58.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:58.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:58.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:58.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:21:59.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:59.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:21:59.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:21:59.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:00.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:00.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:00.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:00.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:01.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:01.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:01.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:01.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:01.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:01.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:01.779 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:22:01.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:01.778717+0000) 2019-10-18T02:22:02.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:22:02.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:02.146781+0000) 2019-10-18T02:22:02.291 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:22:02.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:02.286680+0000) 2019-10-18T02:22:02.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:02.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:02.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:02.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:03.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:03.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:03.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:03.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:04.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:04.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:04.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:04.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:05.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:05.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:05.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:05.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:06.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:06.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:06.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:06.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:06.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:06.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:07.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:07.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:07.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:07.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:07.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:22:07.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:07.692520+0000) 2019-10-18T02:22:08.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:08.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:08.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:08.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:09.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:09.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:09.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:09.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:10.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:10.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:10.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:10.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:11.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:11.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:11.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:11.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:11.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:11.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:11.779 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:22:11.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:11.778905+0000) 2019-10-18T02:22:12.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:22:12.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:12.146889+0000) 2019-10-18T02:22:12.287 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:22:12.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:12.286861+0000) 2019-10-18T02:22:12.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:12.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:12.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:12.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:13.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:13.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:13.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:13.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:14.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:14.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:14.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:14.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:15.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:15.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:15.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:15.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:16.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:16.053+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:16.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:16.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:16.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:16.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:17.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:17.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:17.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:17.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:17.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:22:17.685+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:17.692630+0000) 2019-10-18T02:22:18.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:18.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:18.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:18.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:19.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:19.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:19.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:19.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:20.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:20.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:20.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:20.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:21.059 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:21.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:21.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:21.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:21.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:21.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:21.780 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:22:21.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:21.779098+0000) 2019-10-18T02:22:22.147 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:22:22.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:22.146997+0000) 2019-10-18T02:22:22.288 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:22:22.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:22.287037+0000) 2019-10-18T02:22:22.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:22.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:22.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:22.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:23.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:23.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:23.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:23.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:24.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:24.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:24.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:24.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:25.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:25.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:26.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:26.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:26.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:26.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:26.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:26.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:27.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:27.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:27.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:27.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:27.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:22:27.689+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:27.692734+0000) 2019-10-18T02:22:28.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:28.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:28.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:28.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:29.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:29.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:29.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:29.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:30.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:30.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:30.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:30.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:31.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:31.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:31.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:31.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:31.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:31.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:31.780 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:22:31.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:31.779283+0000) 2019-10-18T02:22:32.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:22:32.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:32.147106+0000) 2019-10-18T02:22:32.288 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:22:32.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:32.287222+0000) 2019-10-18T02:22:32.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:32.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:32.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:32.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:33.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:33.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:33.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:33.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:34.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:34.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:34.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:34.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:35.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:35.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:35.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:35.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:36.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:36.056+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:36.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:36.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:36.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:36.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:37.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:37.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:37.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:37.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:37.693 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:22:37.688+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:37.692837+0000) 2019-10-18T02:22:38.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:38.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:38.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:38.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:39.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:39.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:39.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:39.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:40.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:40.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:40.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:40.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:41.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:41.060+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:41.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:41.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:41.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:41.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:41.780 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:22:41.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:41.779491+0000) 2019-10-18T02:22:42.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:22:42.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:42.147214+0000) 2019-10-18T02:22:42.288 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:22:42.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:42.287408+0000) 2019-10-18T02:22:42.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:42.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:42.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:42.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:43.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:43.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:43.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:43.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:44.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:44.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:44.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:44.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:45.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:45.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:45.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:45.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:46.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:46.060+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:46.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:46.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:46.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:46.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:47.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:47.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:47.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:47.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:47.694 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:22:47.692+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:47.693470+0000) 2019-10-18T02:22:48.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:48.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:48.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:48.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:49.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:49.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:49.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:49.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:50.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:50.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:50.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:50.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:51.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:51.060+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:51.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:51.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:51.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:51.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:51.780 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:22:51.774+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:51.779671+0000) 2019-10-18T02:22:52.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:22:52.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:52.147321+0000) 2019-10-18T02:22:52.288 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:22:52.282+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:52.287587+0000) 2019-10-18T02:22:52.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:52.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:52.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:52.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:53.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:53.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:53.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:53.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:54.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:54.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:54.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:54.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:55.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:55.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:55.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:55.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:56.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:22:56.060+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:22:56.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:56.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:56.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:56.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:57.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:57.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:57.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:57.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:57.694 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:22:57.692+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:22:57.693571+0000) 2019-10-18T02:22:58.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:58.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:58.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:58.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:22:59.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:59.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:22:59.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:22:59.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:00.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:00.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:00.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:00.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:01.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:01.064+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:01.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:01.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:01.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:01.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:01.780 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:23:01.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:01.779889+0000) 2019-10-18T02:23:02.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:23:02.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:02.147425+0000) 2019-10-18T02:23:02.288 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:02.286+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:02.287714+0000) 2019-10-18T02:23:02.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:02.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:02.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:02.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:03.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:03.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:03.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:03.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:04.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:04.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:04.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:04.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:05.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:05.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:05.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:05.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:06.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:06.064+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:06.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:06.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:06.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:06.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:07.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:07.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:07.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:07.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:07.694 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:23:07.692+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:07.693675+0000) 2019-10-18T02:23:08.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:08.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:08.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:08.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:09.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:09.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:10.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:10.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:10.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:10.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:11.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:11.064+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:11.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:11.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:11.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:11.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:11.781 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:23:11.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:11.780072+0000) 2019-10-18T02:23:12.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:23:12.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:12.147536+0000) 2019-10-18T02:23:12.299 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:12.285+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:12.287879+0000) 2019-10-18T02:23:12.299 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:12.285+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:12.288884+0000) 2019-10-18T02:23:12.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:12.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:12.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:12.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:13.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:13.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:13.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:13.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:14.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:14.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:14.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:14.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:15.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:15.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:15.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:15.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:16.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:16.063+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:16.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:16.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:16.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:16.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:16.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:16.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:16.788309+0000) 2019-10-18T02:23:17.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:17.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:17.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:17.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:17.694 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:23:17.691+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:17.693771+0000) 2019-10-18T02:23:18.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:18.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:18.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:18.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:19.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:19.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:20.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:20.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:20.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:20.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:21.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:21.063+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:21.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:21.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:21.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:21.781 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:23:21.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:21.780291+0000) 2019-10-18T02:23:22.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:23:22.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:22.147751+0000) 2019-10-18T02:23:22.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:22.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:22.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:22.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:23.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:23.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:23.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:23.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:24.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:24.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:24.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:24.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:25.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:25.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:26.071 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:26.067+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:26.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:26.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:26.789 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:26.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:26.788497+0000) 2019-10-18T02:23:27.694 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:23:27.691+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:27.693872+0000) 2019-10-18T02:23:27.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:27.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:27.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:27.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:28.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:28.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:28.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:28.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:29.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:29.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:29.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:29.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:30.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:30.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:30.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:30.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:31.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:31.067+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:31.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:31.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:31.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:31.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:31.781 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:23:31.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:31.780509+0000) 2019-10-18T02:23:32.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:23:32.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:32.147853+0000) 2019-10-18T02:23:32.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:32.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:32.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:32.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:33.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:33.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:33.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:33.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:34.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:34.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:34.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:34.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:35.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:35.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:35.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:35.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:36.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:36.067+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:36.789 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:36.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:36.788677+0000) 2019-10-18T02:23:36.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:36.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:36.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:36.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:37.695 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:23:37.691+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:37.693974+0000) 2019-10-18T02:23:37.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:37.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:37.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:37.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:38.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:38.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:38.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:38.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:39.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:39.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:39.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:39.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:40.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:40.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:40.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:40.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:41.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:41.067+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:41.781 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:23:41.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:41.780695+0000) 2019-10-18T02:23:41.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:41.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:41.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:41.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:42.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:23:42.143+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:42.147960+0000) 2019-10-18T02:23:42.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:42.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:42.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:42.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:43.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:43.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:43.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:43.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:44.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:44.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:44.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:44.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:45.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:45.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:45.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:45.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:46.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:46.067+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:46.789 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:46.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:46.788860+0000) 2019-10-18T02:23:46.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:46.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:46.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:46.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:47.695 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:23:47.690+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:47.694085+0000) 2019-10-18T02:23:48.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:48.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:48.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:48.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:49.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:49.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:49.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:49.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:50.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:50.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:50.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:50.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:51.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:51.070+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:51.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:51.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:51.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:51.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:51.781 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:23:51.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:51.780875+0000) 2019-10-18T02:23:52.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:52.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:52.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:52.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:52.148 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:23:52.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:52.148065+0000) 2019-10-18T02:23:53.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:53.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:53.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:53.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:54.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:54.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:54.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:54.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:55.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:55.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:56.076 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:23:56.070+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:23:56.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:56.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:56.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:56.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:56.790 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:23:56.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:56.789103+0000) 2019-10-18T02:23:57.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:57.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:57.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:57.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:57.694 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:23:57.690+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:23:57.694188+0000) 2019-10-18T02:23:58.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:58.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:58.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:58.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:23:59.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:59.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:23:59.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:23:59.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:00.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:00.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:00.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:00.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:01.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:01.070+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:01.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:01.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:01.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:01.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:01.782 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:24:01.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:01.781059+0000) 2019-10-18T02:24:02.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:02.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:02.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:02.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:02.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:24:02.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:02.148169+0000) 2019-10-18T02:24:03.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:03.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:03.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:03.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:04.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:04.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:04.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:04.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:05.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:05.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:06.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:06.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:06.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:06.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:06.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:06.070+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:06.790 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:24:06.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:06.789281+0000) 2019-10-18T02:24:07.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:07.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:07.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:07.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:07.695 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:24:07.690+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:07.694312+0000) 2019-10-18T02:24:08.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:08.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:08.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:08.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:09.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:09.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:09.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:09.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:10.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:10.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:10.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:10.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:11.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:11.074+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:11.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:11.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:11.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:11.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:11.782 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:24:11.777+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:11.781242+0000) 2019-10-18T02:24:12.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:12.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:12.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:12.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:12.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:24:12.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:12.148278+0000) 2019-10-18T02:24:13.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:13.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:13.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:13.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:14.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:14.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:14.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:14.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:15.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:15.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:15.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:15.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:16.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:16.074+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:16.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:16.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:16.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:16.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:16.790 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:24:16.785+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:16.789483+0000) 2019-10-18T02:24:17.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:17.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:17.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:17.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:17.695 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:24:17.690+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:17.694412+0000) 2019-10-18T02:24:18.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:18.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:18.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:18.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:19.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:19.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:19.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:19.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:20.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:20.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:20.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:20.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:21.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:21.074+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:21.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:21.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:21.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:21.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:21.782 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:24:21.776+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:21.781428+0000) 2019-10-18T02:24:22.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:24:22.142+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:22.148381+0000) 2019-10-18T02:24:22.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:22.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:22.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:22.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:23.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:23.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:23.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:23.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:24.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:24.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:24.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:24.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:25.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:25.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:25.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:25.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:26.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:26.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:26.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:26.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:26.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:26.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:26.790 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:24:26.784+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:26.789693+0000) 2019-10-18T02:24:27.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:27.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:27.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:27.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:27.695 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:24:27.689+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:27.694519+0000) 2019-10-18T02:24:28.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:28.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:28.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:28.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:29.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:29.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:29.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:29.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:30.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:30.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:30.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:30.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:31.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:31.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:31.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:31.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:31.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:31.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:31.782 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:24:31.776+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:31.781640+0000) 2019-10-18T02:24:32.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:24:32.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:32.148478+0000) 2019-10-18T02:24:32.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:32.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:32.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:32.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:33.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:33.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:33.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:33.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:34.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:34.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:34.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:34.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:35.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:35.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:35.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:35.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:36.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:36.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:36.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:36.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:36.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:36.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:36.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:24:36.784+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:36.789868+0000) 2019-10-18T02:24:37.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:37.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:37.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:37.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:37.695 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:24:37.689+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:37.694620+0000) 2019-10-18T02:24:38.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:38.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:38.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:38.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:39.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:39.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:39.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:39.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:40.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:40.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:40.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:40.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:41.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:41.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:41.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:41.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:41.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:41.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:41.782 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:24:41.776+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:41.781827+0000) 2019-10-18T02:24:41.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:41.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:41.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:41.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:42.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:24:42.141+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:42.148592+0000) 2019-10-18T02:24:42.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:42.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:42.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:42.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:43.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:43.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:43.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:43.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:44.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:44.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:44.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:44.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:45.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:45.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:45.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:45.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:46.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:46.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:46.790 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:24:46.784+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:46.790046+0000) 2019-10-18T02:24:46.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:46.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:46.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:46.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:47.697 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:24:47.693+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:47.694717+0000) 2019-10-18T02:24:47.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:47.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:47.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:47.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:48.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:48.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:48.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:48.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:49.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:49.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:49.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:49.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:50.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:50.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:50.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:50.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:51.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:51.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:51.783 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:24:51.776+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:51.782008+0000) 2019-10-18T02:24:51.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:51.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:51.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:51.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:52.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:24:52.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:52.148700+0000) 2019-10-18T02:24:52.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:52.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:52.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:52.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:53.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:53.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:53.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:53.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:54.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:54.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:54.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:54.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:55.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:55.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:55.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:55.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:56.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:24:56.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:24:56.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:24:56.788+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:56.790217+0000) 2019-10-18T02:24:56.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:56.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:56.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:56.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:57.696 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:24:57.693+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:24:57.694822+0000) 2019-10-18T02:24:57.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:57.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:57.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:57.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:58.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:58.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:58.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:58.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:24:59.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:59.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:24:59.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:24:59.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:00.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:00.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:00.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:00.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:01.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:01.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:01.783 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:25:01.780+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:01.782215+0000) 2019-10-18T02:25:01.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:01.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:01.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:01.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:02.149 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:25:02.144+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:02.148805+0000) 2019-10-18T02:25:02.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:02.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:02.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:02.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:03.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:03.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:03.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:03.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:04.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:04.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:04.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:04.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:05.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:05.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:05.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:05.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:06.088 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:06.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:06.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:06.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:06.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:06.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:06.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:25:06.788+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:06.790362+0000) 2019-10-18T02:25:07.698 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:25:07.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:07.697539+0000) 2019-10-18T02:25:07.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:07.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:07.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:07.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:08.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:08.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:08.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:08.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:09.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:09.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:09.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:09.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:10.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:10.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:10.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:10.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:11.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:11.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:11.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:11.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:11.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:11.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:11.783 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:25:11.780+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:11.782365+0000) 2019-10-18T02:25:12.150 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:25:12.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:12.149471+0000) 2019-10-18T02:25:12.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:12.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:12.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:12.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:13.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:13.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:13.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:13.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:14.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:14.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:14.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:14.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:15.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:15.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:15.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:15.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:16.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:16.084+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:16.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:16.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:16.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:25:16.788+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:16.790543+0000) 2019-10-18T02:25:17.698 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:25:17.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:17.697639+0000) 2019-10-18T02:25:17.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:17.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:17.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:17.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:18.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:18.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:18.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:18.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:19.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:19.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:20.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:20.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:20.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:20.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:21.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:21.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:21.783 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:25:21.780+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:21.782517+0000) 2019-10-18T02:25:21.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:21.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:21.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:21.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:22.150 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:25:22.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:22.149849+0000) 2019-10-18T02:25:22.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:22.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:22.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:22.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:23.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:23.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:23.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:23.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:24.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:24.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:24.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:24.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:25.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:25.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:25.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:25.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:26.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:26.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:26.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:25:26.788+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:26.790712+0000) 2019-10-18T02:25:26.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:26.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:26.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:26.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:27.698 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:25:27.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:27.697739+0000) 2019-10-18T02:25:27.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:27.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:27.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:27.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:28.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:28.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:28.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:28.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:29.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:29.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:29.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:29.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:30.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:30.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:30.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:30.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:31.092 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:31.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:31.783 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:25:31.780+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:31.782721+0000) 2019-10-18T02:25:31.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:31.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:31.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:31.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:32.150 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:25:32.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:32.149954+0000) 2019-10-18T02:25:32.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:32.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:32.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:32.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:33.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:33.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:33.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:33.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:34.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:34.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:34.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:34.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:35.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:35.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:35.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:35.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:36.092 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:36.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:36.791 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:25:36.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:36.790887+0000) 2019-10-18T02:25:37.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:36.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:37.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:36.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:37.698 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:25:37.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:37.697838+0000) 2019-10-18T02:25:38.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:38.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:38.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:38.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:39.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:39.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:39.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:39.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:40.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:40.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:40.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:40.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:41.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:41.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:41.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:41.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:41.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:41.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:41.783 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:25:41.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:41.782911+0000) 2019-10-18T02:25:42.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:42.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:42.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:42.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:42.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:25:42.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:42.150069+0000) 2019-10-18T02:25:43.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:43.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:43.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:43.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:44.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:44.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:44.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:44.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:45.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:45.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:45.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:45.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:46.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:46.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:46.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:46.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:46.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:46.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:46.792 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:25:46.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:46.791060+0000) 2019-10-18T02:25:47.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:47.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:47.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:47.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:47.698 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:25:47.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:47.697938+0000) 2019-10-18T02:25:48.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:48.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:48.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:48.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:49.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:49.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:49.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:49.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:50.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:50.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:50.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:50.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:51.095 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:51.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:51.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:51.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:51.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:51.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:51.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:25:51.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:51.783093+0000) 2019-10-18T02:25:52.151 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:25:52.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:52.150180+0000) 2019-10-18T02:25:52.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:52.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:52.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:52.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:53.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:53.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:53.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:53.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:54.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:54.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:54.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:54.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:55.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:55.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:55.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:55.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:56.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:25:56.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:25:56.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:56.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:56.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:56.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:56.792 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:25:56.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:56.791250+0000) 2019-10-18T02:25:57.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:57.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:57.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:57.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:57.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:25:57.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:25:57.698037+0000) 2019-10-18T02:25:58.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:58.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:58.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:58.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:25:59.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:59.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:25:59.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:25:59.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:00.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:00.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:00.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:00.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:01.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:01.091+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:01.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:01.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:01.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:01.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:01.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:26:01.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:01.783305+0000) 2019-10-18T02:26:02.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:26:02.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:02.151480+0000) 2019-10-18T02:26:02.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:02.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:02.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:02.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:03.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:03.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:03.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:03.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:04.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:04.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:04.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:04.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:05.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:05.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:05.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:05.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:06.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:06.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:06.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:06.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:06.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:06.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:06.792 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:26:06.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:06.791436+0000) 2019-10-18T02:26:07.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:07.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:07.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:07.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:07.698 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:26:07.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:07.698141+0000) 2019-10-18T02:26:08.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:08.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:08.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:08.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:09.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:09.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:09.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:09.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:10.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:10.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:10.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:10.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:11.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:11.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:11.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:11.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:11.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:11.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:11.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:26:11.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:11.783490+0000) 2019-10-18T02:26:12.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:26:12.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:12.151590+0000) 2019-10-18T02:26:12.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:12.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:12.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:12.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:13.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:13.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:13.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:13.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:14.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:14.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:14.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:14.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:15.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:15.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:15.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:15.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:16.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:16.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:16.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:16.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:16.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:16.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:16.792 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:26:16.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:16.791649+0000) 2019-10-18T02:26:17.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:17.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:17.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:17.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:17.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:26:17.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:17.698246+0000) 2019-10-18T02:26:18.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:18.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:18.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:18.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:19.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:19.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:19.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:19.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:20.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:20.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:20.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:20.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:21.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:21.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:21.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:21.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:21.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:21.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:21.784 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:26:21.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:21.783670+0000) 2019-10-18T02:26:22.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:22.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:22.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:22.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:22.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:26:22.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:22.151697+0000) 2019-10-18T02:26:23.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:23.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:23.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:23.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:24.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:24.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:24.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:24.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:25.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:25.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:25.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:25.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:26.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:26.094+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:26.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:26.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:26.792 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:26:26.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:26.791811+0000) 2019-10-18T02:26:27.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:27.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:27.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:27.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:27.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:26:27.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:27.698348+0000) 2019-10-18T02:26:28.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:28.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:28.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:28.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:29.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:29.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:29.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:29.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:30.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:30.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:30.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:30.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:31.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:31.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:31.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:31.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:31.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:31.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:31.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:26:31.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:31.783860+0000) 2019-10-18T02:26:32.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:26:32.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:32.151798+0000) 2019-10-18T02:26:32.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:32.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:32.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:32.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:33.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:33.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:33.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:33.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:34.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:34.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:34.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:34.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:35.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:35.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:35.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:35.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:36.103 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:36.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:36.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:36.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:36.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:36.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:36.794 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:26:36.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:36.791991+0000) 2019-10-18T02:26:37.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:37.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:37.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:37.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:37.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:26:37.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:37.698450+0000) 2019-10-18T02:26:38.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:38.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:38.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:38.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:39.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:39.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:39.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:39.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:40.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:40.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:40.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:40.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:41.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:41.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:41.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:41.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:41.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:41.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:41.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:26:41.779+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:41.784102+0000) 2019-10-18T02:26:42.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:42.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:42.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:42.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:42.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:26:42.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:42.151912+0000) 2019-10-18T02:26:43.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:43.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:43.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:43.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:44.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:44.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:44.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:44.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:45.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:45.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:45.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:45.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:46.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:46.097+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:46.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:46.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:46.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:46.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:46.793 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:26:46.787+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:46.792167+0000) 2019-10-18T02:26:47.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:47.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:47.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:47.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:47.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:26:47.693+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:47.698550+0000) 2019-10-18T02:26:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:48.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:48.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:49.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:49.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:49.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:49.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:50.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:50.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:50.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:50.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:51.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:51.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:51.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:51.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:51.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:51.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:51.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:26:51.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:51.784297+0000) 2019-10-18T02:26:52.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:26:52.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:52.152012+0000) 2019-10-18T02:26:52.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:52.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:52.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:52.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:53.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:53.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:54.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:54.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:54.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:54.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:55.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:55.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:55.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:55.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:56.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:26:56.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:26:56.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:56.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:56.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:56.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:56.793 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:26:56.786+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:56.792355+0000) 2019-10-18T02:26:57.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:57.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:57.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:57.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:57.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:26:57.693+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:26:57.698650+0000) 2019-10-18T02:26:58.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:58.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:58.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:58.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:26:59.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:59.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:26:59.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:26:59.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:00.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:00.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:00.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:00.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:01.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:01.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:01.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:01.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:01.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:01.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:01.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:27:01.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:01.784499+0000) 2019-10-18T02:27:02.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:02.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:02.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:02.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:02.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:27:02.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:02.152109+0000) 2019-10-18T02:27:03.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:03.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:03.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:03.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:04.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:04.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:04.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:04.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:05.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:05.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:05.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:05.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:06.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:06.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:06.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:06.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:06.108 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:06.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:06.793 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:27:06.786+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:06.792542+0000) 2019-10-18T02:27:07.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:07.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:07.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:07.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:07.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:27:07.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:07.698750+0000) 2019-10-18T02:27:08.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:08.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:08.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:08.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:09.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:09.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:09.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:09.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:10.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:10.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:10.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:10.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:11.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:11.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:11.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:11.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:11.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:11.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:11.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:27:11.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:11.784684+0000) 2019-10-18T02:27:12.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:27:12.149+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:12.152217+0000) 2019-10-18T02:27:12.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:12.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:12.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:12.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:13.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:13.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:13.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:13.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:14.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:14.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:14.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:14.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:15.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:15.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:15.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:15.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:16.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:16.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:16.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:16.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:16.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:16.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:16.793 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:27:16.786+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:16.792763+0000) 2019-10-18T02:27:17.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:17.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:17.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:17.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:17.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:27:17.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:17.698853+0000) 2019-10-18T02:27:18.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:18.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:18.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:18.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:19.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:19.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:19.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:19.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:20.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:20.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:20.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:20.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:21.111 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:21.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:21.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:21.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:21.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:21.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:21.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:27:21.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:21.784869+0000) 2019-10-18T02:27:22.152 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:27:22.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:22.152317+0000) 2019-10-18T02:27:22.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:22.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:22.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:22.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:23.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:23.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:23.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:23.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:24.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:24.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:24.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:24.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:25.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:25.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:25.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:25.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:26.112 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:26.108+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:26.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:26.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:26.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:26.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:26.793 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:27:26.786+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:26.792944+0000) 2019-10-18T02:27:27.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:27.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:27.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:27.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:27.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:27:27.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:27.698958+0000) 2019-10-18T02:27:28.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:28.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:28.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:28.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:29.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:29.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:29.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:29.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:30.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:30.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:30.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:30.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:31.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:31.108+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:31.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:31.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:31.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:31.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:31.785 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:27:31.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:31.785064+0000) 2019-10-18T02:27:32.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:27:32.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:32.152415+0000) 2019-10-18T02:27:32.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:32.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:32.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:32.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:33.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:33.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:33.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:33.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:34.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:34.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:34.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:34.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:35.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:35.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:35.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:35.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:36.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:36.108+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:36.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:36.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:36.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:36.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:36.794 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:27:36.786+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:36.793111+0000) 2019-10-18T02:27:37.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:37.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:37.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:37.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:37.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:27:37.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:37.699059+0000) 2019-10-18T02:27:38.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:38.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:38.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:38.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:39.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:39.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:39.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:39.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:40.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:40.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:40.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:40.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:41.114 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:41.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:41.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:41.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:41.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:41.786 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:27:41.778+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:41.785261+0000) 2019-10-18T02:27:42.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:27:42.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:42.152520+0000) 2019-10-18T02:27:42.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:42.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:42.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:42.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:43.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:43.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:43.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:43.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:44.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:44.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:44.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:44.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:45.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:45.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:45.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:45.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:46.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:46.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:46.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:46.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:46.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:46.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:46.794 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:27:46.786+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:46.793289+0000) 2019-10-18T02:27:47.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:47.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:47.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:47.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:47.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:27:47.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:47.699161+0000) 2019-10-18T02:27:48.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:48.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:48.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:48.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:49.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:49.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:49.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:49.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:50.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:50.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:50.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:50.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:51.116 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:51.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:51.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:51.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:51.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:51.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:51.786 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:27:51.782+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:51.785460+0000) 2019-10-18T02:27:52.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:52.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:52.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:52.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:52.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:27:52.148+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:52.152621+0000) 2019-10-18T02:27:53.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:53.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:53.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:53.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:54.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:54.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:54.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:54.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:55.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:55.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:55.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:55.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:56.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:27:56.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:27:56.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:56.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:56.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:56.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:56.794 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:27:56.790+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:56.793506+0000) 2019-10-18T02:27:57.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:57.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:57.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:57.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:57.699 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:27:57.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:27:57.699262+0000) 2019-10-18T02:27:58.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:58.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:58.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:58.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:27:59.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:59.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:27:59.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:27:59.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:00.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:00.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:00.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:00.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:01.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:01.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:01.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:01.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:01.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:01.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:01.786 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:28:01.781+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:01.785641+0000) 2019-10-18T02:28:02.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:28:02.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:02.152716+0000) 2019-10-18T02:28:02.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:02.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:02.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:02.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:03.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:03.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:03.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:03.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:04.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:04.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:04.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:04.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:05.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:05.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:05.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:05.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:06.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:06.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:06.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:06.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:06.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:06.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:06.794 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:28:06.789+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:06.793726+0000) 2019-10-18T02:28:07.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:07.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:07.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:07.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:07.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:28:07.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:07.699363+0000) 2019-10-18T02:28:08.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:08.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:08.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:08.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:09.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:09.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:09.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:09.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:10.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:10.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:10.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:10.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:11.120 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:11.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:11.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:11.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:11.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:11.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:11.786 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:28:11.781+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:11.785833+0000) 2019-10-18T02:28:12.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:28:12.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:12.152817+0000) 2019-10-18T02:28:12.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:12.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:12.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:12.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:13.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:13.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:13.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:13.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:14.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:14.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:14.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:14.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:15.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:15.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:15.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:15.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:16.120 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:16.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:16.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:16.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:16.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:16.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:16.794 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:28:16.789+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:16.793935+0000) 2019-10-18T02:28:17.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:17.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:17.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:17.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:17.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:28:17.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:17.699467+0000) 2019-10-18T02:28:18.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:18.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:18.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:18.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:19.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:19.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:19.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:19.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:20.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:20.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:20.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:20.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:21.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:21.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:21.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:21.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:21.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:21.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:21.786 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:28:21.781+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:21.786020+0000) 2019-10-18T02:28:22.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:28:22.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:22.152917+0000) 2019-10-18T02:28:22.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:22.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:22.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:22.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:23.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:23.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:23.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:23.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:24.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:24.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:24.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:24.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:25.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:25.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:25.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:25.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:26.122 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:26.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:26.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:26.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:26.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:26.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:26.795 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:28:26.789+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:26.794147+0000) 2019-10-18T02:28:27.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:27.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:27.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:27.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:27.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:28:27.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:27.699586+0000) 2019-10-18T02:28:28.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:28.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:28.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:28.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:29.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:29.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:29.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:29.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:30.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:30.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:31.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:31.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:31.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:31.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:31.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:31.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:31.787 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:28:31.785+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:31.786215+0000) 2019-10-18T02:28:32.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:28:32.147+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:32.153015+0000) 2019-10-18T02:28:32.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:32.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:32.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:32.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:33.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:33.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:33.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:33.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:34.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:34.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:34.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:34.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:35.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:35.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:35.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:35.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:36.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:36.118+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:36.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:36.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:36.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:36.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:36.795 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:28:36.793+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:36.794336+0000) 2019-10-18T02:28:37.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:37.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:37.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:37.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:37.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:28:37.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:37.699686+0000) 2019-10-18T02:28:38.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:38.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:38.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:38.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:39.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:39.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:39.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:39.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:40.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:40.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:40.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:40.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:41.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:41.118+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:41.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:41.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:41.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:41.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:41.787 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:28:41.785+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:41.786378+0000) 2019-10-18T02:28:42.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:28:42.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:42.153114+0000) 2019-10-18T02:28:42.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:42.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:42.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:42.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:43.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:43.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:43.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:43.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:44.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:44.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:44.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:44.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:45.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:45.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:45.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:45.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:46.126 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:46.118+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:46.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:46.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:46.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:46.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:46.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:28:46.793+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:46.794508+0000) 2019-10-18T02:28:47.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:47.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:47.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:47.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:47.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:28:47.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:47.699786+0000) 2019-10-18T02:28:48.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:48.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:48.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:48.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:49.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:49.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:49.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:49.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:50.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:50.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:50.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:50.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:51.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:51.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:51.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:51.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:51.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:51.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:51.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:28:51.785+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:51.786532+0000) 2019-10-18T02:28:52.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:28:52.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:52.153210+0000) 2019-10-18T02:28:52.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:52.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:18:19.671371+0000 (oldest deadline 2019-10-18T02:18:39.671371+0000) 2019-10-18T02:28:52.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:52.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:53.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:53.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:54.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:54.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:55.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:55.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:56.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:28:56.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:28:56.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:56.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:56.795 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:28:56.793+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:56.794698+0000) 2019-10-18T02:28:57.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:57.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:57.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:28:57.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:28:57.699890+0000) 2019-10-18T02:28:58.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:58.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:28:59.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:28:59.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:00.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:00.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:01.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:01.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:01.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:01.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:01.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:29:01.785+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:01.786699+0000) 2019-10-18T02:29:02.153 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:29:02.146+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:02.153306+0000) 2019-10-18T02:29:02.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:02.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:03.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:03.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:04.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:04.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:05.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:05.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:06.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:06.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:06.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:06.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:06.795 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:29:06.793+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:06.794877+0000) 2019-10-18T02:29:07.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:07.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:07.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:29:07.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:07.699996+0000) 2019-10-18T02:29:08.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:08.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:09.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:09.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:10.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:10.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:11.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:11.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:11.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:11.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:11.787 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:29:11.785+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:11.786881+0000) 2019-10-18T02:29:12.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:29:12.145+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:12.153409+0000) 2019-10-18T02:29:12.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:12.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:13.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:13.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:14.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:14.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:14.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:14.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:15.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:15.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:15.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:15.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:16.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:16.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:16.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:16.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:16.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:16.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:16.796 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:29:16.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:16.795067+0000) 2019-10-18T02:29:17.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:17.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:17.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:17.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:17.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:29:17.693+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:17.700105+0000) 2019-10-18T02:29:18.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:18.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:18.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:18.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:19.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:19.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:19.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:19.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:20.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:20.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:20.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:20.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:21.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:21.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:21.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:21.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:21.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:21.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:21.787 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:29:21.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:21.787100+0000) 2019-10-18T02:29:22.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:29:22.149+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:22.153526+0000) 2019-10-18T02:29:22.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:22.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:22.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:22.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:23.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:23.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:23.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:23.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:24.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:24.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:24.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:24.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:25.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:25.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:25.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:25.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:26.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:26.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:26.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:26.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:26.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:26.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:26.796 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:29:26.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:26.795280+0000) 2019-10-18T02:29:27.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:27.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:27.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:27.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:27.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:29:27.693+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:27.700205+0000) 2019-10-18T02:29:28.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:28.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:28.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:28.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:29.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:29.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:29.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:29.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:30.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:30.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:30.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:30.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:31.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:31.129+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:31.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:31.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:31.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:31.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:31.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:29:31.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:31.787286+0000) 2019-10-18T02:29:32.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:29:32.153+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:32.153621+0000) 2019-10-18T02:29:32.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:32.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:32.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:32.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:33.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:33.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:33.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:33.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:34.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:34.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:34.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:34.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:35.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:35.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:35.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:35.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:36.134 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:36.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:36.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:36.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:36.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:36.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:36.796 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:29:36.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:36.795493+0000) 2019-10-18T02:29:37.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:37.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:37.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:37.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:37.700 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:29:37.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:37.700306+0000) 2019-10-18T02:29:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:38.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:38.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:39.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:39.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:39.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:39.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:40.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:40.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:40.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:40.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:41.135 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:41.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:41.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:41.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:41.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:41.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:41.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:29:41.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:41.787486+0000) 2019-10-18T02:29:42.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:29:42.153+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:42.153718+0000) 2019-10-18T02:29:42.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:42.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:42.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:42.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:43.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:43.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:43.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:43.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:44.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:44.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:44.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:44.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:45.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:45.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:45.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:45.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:46.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:46.132+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:46.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:46.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:46.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:46.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:46.796 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:29:46.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:46.795703+0000) 2019-10-18T02:29:47.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:47.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:47.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:47.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:47.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:29:47.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:47.700417+0000) 2019-10-18T02:29:48.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:48.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:48.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:48.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:49.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:49.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:49.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:49.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:50.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:50.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:50.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:50.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:51.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:51.132+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:51.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:51.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:51.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:51.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:51.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:29:51.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:51.787692+0000) 2019-10-18T02:29:52.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:29:52.152+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:52.153822+0000) 2019-10-18T02:29:52.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:52.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:52.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:52.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:53.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:53.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:53.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:53.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:54.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:54.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:54.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:54.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:55.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:55.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:55.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:55.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:56.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:29:56.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:29:56.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:56.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:56.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:56.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:56.796 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:29:56.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:56.795878+0000) 2019-10-18T02:29:57.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:57.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:57.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:57.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:57.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:29:57.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:29:57.700517+0000) 2019-10-18T02:29:58.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:58.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:58.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:58.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:29:59.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:59.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:29:59.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:29:59.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:00.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:00.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:00.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:00.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:01.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:01.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:01.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:01.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:01.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:01.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:01.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:30:01.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:01.787964+0000) 2019-10-18T02:30:02.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:02.152+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:02.153923+0000) 2019-10-18T02:30:02.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:02.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:02.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:02.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:03.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:03.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:03.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:03.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:04.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:04.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:04.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:04.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:05.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:05.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:05.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:05.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:06.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:06.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:06.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:06.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:06.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:06.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:06.796 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:30:06.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:06.796056+0000) 2019-10-18T02:30:07.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:07.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:07.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:07.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:07.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:07.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:07.700624+0000) 2019-10-18T02:30:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:08.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:08.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:09.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:09.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:09.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:09.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:10.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:10.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:10.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:10.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:11.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:11.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:11.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:11.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:11.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:11.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:11.788 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:30:11.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:11.788187+0000) 2019-10-18T02:30:12.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:12.152+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:12.154022+0000) 2019-10-18T02:30:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:12.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:12.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:13.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:13.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:13.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:13.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:14.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:14.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:14.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:14.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:15.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:15.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:15.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:15.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:16.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:16.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:16.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:16.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:16.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:16.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:16.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:30:16.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:16.796243+0000) 2019-10-18T02:30:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:17.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:17.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:17.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:17.696+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:17.700731+0000) 2019-10-18T02:30:18.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:18.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:18.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:18.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:19.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:19.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:19.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:19.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:20.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:20.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:20.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:20.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:21.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:21.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:21.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:21.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:21.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:21.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:21.789 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:30:21.784+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:21.788421+0000) 2019-10-18T02:30:22.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:22.151+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:22.154125+0000) 2019-10-18T02:30:22.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:22.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:22.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:22.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:23.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:23.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:23.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:23.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:24.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:24.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:24.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:24.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:25.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:25.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:25.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:25.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:26.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:26.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:26.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:26.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:26.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:26.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:26.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:30:26.792+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:26.796435+0000) 2019-10-18T02:30:27.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:27.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:27.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:27.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:27.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:27.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:27.700831+0000) 2019-10-18T02:30:28.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:28.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:28.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:28.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:29.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:29.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:29.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:29.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:30.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:30.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:30.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:30.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:31.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:31.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:31.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:31.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:31.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:31.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:31.789 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:30:31.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:31.788641+0000) 2019-10-18T02:30:32.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:32.151+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:32.154223+0000) 2019-10-18T02:30:32.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:32.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:32.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:32.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:33.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:33.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:33.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:33.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:34.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:34.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:34.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:34.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:35.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:35.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:35.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:35.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:36.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:36.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:36.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:36.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:36.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:36.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:36.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:30:36.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:36.796617+0000) 2019-10-18T02:30:37.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:37.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:37.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:37.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:37.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:37.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:37.700935+0000) 2019-10-18T02:30:38.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:38.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:38.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:38.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:39.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:39.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:39.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:39.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:40.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:40.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:40.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:40.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:41.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:41.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:41.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:41.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:41.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:41.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:41.789 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:30:41.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:41.788911+0000) 2019-10-18T02:30:42.154 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:42.151+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:42.154312+0000) 2019-10-18T02:30:42.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:42.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:42.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:42.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:43.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:43.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:43.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:43.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:44.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:44.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:44.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:44.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:45.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:45.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:45.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:45.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:46.146 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:46.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:46.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:46.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:46.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:46.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:46.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:30:46.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:46.796795+0000) 2019-10-18T02:30:47.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:47.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:47.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:47.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:47.701 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:47.695+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:47.701034+0000) 2019-10-18T02:30:48.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:48.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:48.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:49.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:49.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:49.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:49.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:50.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:50.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:50.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:50.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:51.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:51.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:51.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:51.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:51.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:51.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:51.789 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:30:51.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:51.789175+0000) 2019-10-18T02:30:52.155 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:52.151+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:52.154409+0000) 2019-10-18T02:30:52.156 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:52.151+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:52.156050+0000) 2019-10-18T02:30:52.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:52.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:52.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:52.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:53.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:53.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:53.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:53.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:54.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:54.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:54.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:54.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:55.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:55.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:55.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:55.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:56.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:30:56.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:30:56.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:56.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:56.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:56.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:56.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:30:56.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:56.654599+0000) 2019-10-18T02:30:56.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:30:56.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:56.797006+0000) 2019-10-18T02:30:57.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:57.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:57.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:57.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:57.702 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:57.694+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:57.701133+0000) 2019-10-18T02:30:57.703 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:30:57.698+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:30:57.702211+0000) 2019-10-18T02:30:58.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:58.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:58.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:58.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:30:59.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:59.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:30:59.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:30:59.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:00.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:00.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:00.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:00.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:01.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:01.142+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:01.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:01.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:01.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:01.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:01.790 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:31:01.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:01.789392+0000) 2019-10-18T02:31:02.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:31:02.194+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:02.201346+0000) 2019-10-18T02:31:02.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:02.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:02.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:02.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:03.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:03.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:03.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:04.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:04.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:04.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:04.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:05.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:05.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:05.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:05.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:06.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:06.146+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:06.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:06.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:06.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:06.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:06.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:31:06.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:06.654699+0000) 2019-10-18T02:31:06.797 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:31:06.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:06.797185+0000) 2019-10-18T02:31:07.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:07.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:07.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:07.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:08.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:08.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:08.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:08.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:09.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:09.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:09.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:09.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:10.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:10.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:10.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:10.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:11.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:11.146+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:11.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:11.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:11.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:11.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:11.790 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:31:11.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:11.789650+0000) 2019-10-18T02:31:12.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:12.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:12.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:31:12.198+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:12.201460+0000) 2019-10-18T02:31:12.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:12.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:13.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:13.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:13.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:13.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:14.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:14.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:14.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:14.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:15.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:15.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:15.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:15.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:16.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:16.146+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:16.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:16.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:16.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:16.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:16.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:31:16.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:16.654799+0000) 2019-10-18T02:31:16.798 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:31:16.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:16.797390+0000) 2019-10-18T02:31:17.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:17.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:17.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:17.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:18.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:18.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:18.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:18.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:19.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:19.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:19.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:19.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:20.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:20.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:20.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:20.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:21.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:21.146+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:21.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:21.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:21.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:21.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:21.790 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:31:21.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:21.789868+0000) 2019-10-18T02:31:22.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:31:22.198+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:22.201571+0000) 2019-10-18T02:31:22.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:22.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:22.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:22.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:23.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:23.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:23.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:23.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:24.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:24.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:24.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:24.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:25.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:25.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:25.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:25.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:26.154 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:26.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:26.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:26.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:26.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:26.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:26.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:31:26.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:26.654898+0000) 2019-10-18T02:31:26.798 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:31:26.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:26.797576+0000) 2019-10-18T02:31:27.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:27.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:27.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:27.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:28.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:28.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:28.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:28.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:29.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:29.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:29.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:29.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:30.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:30.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:30.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:30.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:31.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:31.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:31.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:31.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:31.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:31.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:31.790 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:31:31.783+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:31.790115+0000) 2019-10-18T02:31:32.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:31:32.197+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:32.201677+0000) 2019-10-18T02:31:32.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:32.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:32.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:32.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:33.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:33.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:33.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:33.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:34.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:34.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:34.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:35.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:35.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:35.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:35.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:36.158 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:36.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:36.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:36.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:36.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:36.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:36.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:31:36.649+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:36.654994+0000) 2019-10-18T02:31:36.800 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:31:36.791+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:36.797768+0000) 2019-10-18T02:31:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:37.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:37.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:38.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:38.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:38.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:38.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:39.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:39.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:39.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:39.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:40.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:40.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:40.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:40.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:41.159 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:41.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:41.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:41.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:41.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:41.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:41.791 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:31:41.787+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:41.790304+0000) 2019-10-18T02:31:42.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:31:42.197+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:42.201779+0000) 2019-10-18T02:31:42.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:42.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:42.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:42.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:43.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:43.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:43.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:43.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:44.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:44.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:44.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:44.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:45.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:45.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:45.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:45.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:46.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:46.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:46.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:46.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:46.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:46.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:46.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:31:46.649+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:46.655091+0000) 2019-10-18T02:31:46.798 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:31:46.790+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:46.797970+0000) 2019-10-18T02:31:47.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:47.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:47.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:47.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:48.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:48.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:48.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:48.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:49.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:49.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:49.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:49.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:50.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:50.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:50.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:50.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:51.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:51.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:51.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:51.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:51.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:51.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:51.791 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:31:51.786+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:51.790537+0000) 2019-10-18T02:31:52.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:31:52.201+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:52.201883+0000) 2019-10-18T02:31:52.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:52.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:52.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:52.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:53.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:53.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:53.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:53.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:54.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:54.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:54.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:54.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:55.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:55.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:55.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:55.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:56.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:31:56.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:31:56.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:56.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:56.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:56.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:56.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:31:56.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:56.655194+0000) 2019-10-18T02:31:56.798 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:31:56.794+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:31:56.798154+0000) 2019-10-18T02:31:57.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:57.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:57.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:57.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:58.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:58.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:58.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:58.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:31:59.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:59.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:31:59.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:31:59.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:00.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:00.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:00.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:00.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:01.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:01.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:01.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:01.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:01.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:01.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:01.791 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:32:01.786+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:01.790759+0000) 2019-10-18T02:32:02.202 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:02.201+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:02.201973+0000) 2019-10-18T02:32:02.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:02.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:02.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:02.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:03.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:03.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:03.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:03.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:04.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:04.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:04.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:04.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:05.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:05.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:05.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:05.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:06.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:06.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:06.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:06.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:06.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:06.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:06.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:32:06.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:06.655300+0000) 2019-10-18T02:32:06.799 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:32:06.794+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:06.798368+0000) 2019-10-18T02:32:07.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:07.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:07.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:07.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:08.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:08.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:08.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:08.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:09.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:09.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:09.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:09.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:10.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:10.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:10.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:10.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:11.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:11.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:11.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:11.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:11.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:11.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:11.791 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:32:11.786+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:11.790985+0000) 2019-10-18T02:32:12.203 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:12.200+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:12.202064+0000) 2019-10-18T02:32:12.203 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:12.200+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:12.203107+0000) 2019-10-18T02:32:12.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:12.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:12.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:12.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:13.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:13.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:13.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:13.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:14.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:14.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:14.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:14.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:15.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:15.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:15.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:15.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:16.167 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:16.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:16.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:16.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:16.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:16.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:16.655 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:32:16.652+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:16.655397+0000) 2019-10-18T02:32:16.702 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:16.700+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:16.702265+0000) 2019-10-18T02:32:16.799 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:32:16.794+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:16.798553+0000) 2019-10-18T02:32:17.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:17.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:17.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:17.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:18.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:18.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:18.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:18.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:19.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:19.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:19.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:19.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:20.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:20.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:21.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:21.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:21.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:21.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:21.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:21.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:21.791 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:32:21.786+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:21.791218+0000) 2019-10-18T02:32:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:22.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:22.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:23.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:23.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:23.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:23.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:24.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:24.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:24.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:24.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:25.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:25.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:25.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:25.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:26.168 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:26.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:26.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:26.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:26.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:26.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:26.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:32:26.652+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:26.655503+0000) 2019-10-18T02:32:26.702 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:26.700+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:26.702362+0000) 2019-10-18T02:32:26.799 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:32:26.794+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:26.798758+0000) 2019-10-18T02:32:27.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:27.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:27.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:27.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:28.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:28.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:28.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:28.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:29.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:29.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:29.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:29.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:30.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:30.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:30.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:30.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:31.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:31.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:31.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:31.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:31.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:31.792 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:32:31.786+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:31.791454+0000) 2019-10-18T02:32:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:32.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:32.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:33.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:33.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:33.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:33.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:34.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:34.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:34.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:34.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:35.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:35.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:35.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:35.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:36.171 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:36.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:36.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:32:36.652+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:36.655609+0000) 2019-10-18T02:32:36.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:36.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:36.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:36.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:36.702 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:36.700+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:36.702465+0000) 2019-10-18T02:32:36.799 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:32:36.794+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:36.798936+0000) 2019-10-18T02:32:37.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:37.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:37.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:37.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:38.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:38.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:38.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:38.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:39.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:39.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:39.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:39.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:40.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:40.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:40.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:40.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:41.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:41.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:41.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:41.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:41.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:41.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:41.792 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:32:41.786+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:41.791681+0000) 2019-10-18T02:32:42.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:42.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:42.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:42.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:43.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:43.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:43.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:43.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:44.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:44.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:44.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:44.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:45.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:45.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:45.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:45.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:46.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:46.167+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:46.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:46.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:46.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:46.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:46.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:32:46.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:46.655709+0000) 2019-10-18T02:32:46.703 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:46.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:46.702571+0000) 2019-10-18T02:32:46.799 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:32:46.798+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:46.799163+0000) 2019-10-18T02:32:47.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:47.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:47.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:47.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:48.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:48.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:48.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:48.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:49.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:49.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:49.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:49.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:50.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:50.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:50.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:50.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:51.174 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:51.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:51.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:51.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:51.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:51.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:51.792 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:32:51.790+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:51.791936+0000) 2019-10-18T02:32:52.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:52.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:52.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:52.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:53.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:53.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:53.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:53.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:54.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:54.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:54.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:54.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:55.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:55.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:55.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:55.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:56.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:32:56.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:32:56.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:56.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:56.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:56.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:56.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:32:56.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:56.655815+0000) 2019-10-18T02:32:56.703 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:32:56.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:56.702672+0000) 2019-10-18T02:32:56.800 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:32:56.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:32:56.799347+0000) 2019-10-18T02:32:57.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:57.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:57.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:57.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:58.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:58.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:58.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:58.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:32:59.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:59.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:32:59.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:32:59.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:00.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:00.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:01.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:01.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:01.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:01.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:01.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:01.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:01.792 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:33:01.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:01.792174+0000) 2019-10-18T02:33:02.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:02.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:02.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:02.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:03.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:03.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:03.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:03.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:04.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:04.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:04.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:04.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:05.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:05.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:05.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:05.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:06.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:06.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:06.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:33:06.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:06.655918+0000) 2019-10-18T02:33:06.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:06.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:06.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:06.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:06.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:33:06.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:06.703192+0000) 2019-10-18T02:33:06.800 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:33:06.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:06.799532+0000) 2019-10-18T02:33:07.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:07.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:07.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:07.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:08.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:08.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:08.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:08.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:09.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:09.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:09.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:09.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:10.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:10.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:10.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:10.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:11.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:11.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:11.793 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:33:11.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:11.792399+0000) 2019-10-18T02:33:11.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:11.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:11.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:11.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:12.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:12.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:12.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:12.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:13.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:13.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:13.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:13.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:14.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:14.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:14.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:14.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:15.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:15.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:15.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:15.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:16.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:16.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:16.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:33:16.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:16.656015+0000) 2019-10-18T02:33:16.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:16.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:16.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:16.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:16.703 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:33:16.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:16.703294+0000) 2019-10-18T02:33:16.800 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:33:16.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:16.799746+0000) 2019-10-18T02:33:17.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:17.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:17.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:17.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:18.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:18.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:18.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:18.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:19.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:19.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:19.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:19.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:20.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:20.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:20.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:20.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:21.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:21.174+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:21.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:21.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:21.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:21.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:21.793 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:33:21.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:21.792638+0000) 2019-10-18T02:33:22.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:22.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:22.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:22.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:23.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:23.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:23.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:23.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:24.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:24.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:24.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:24.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:25.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:25.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:25.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:25.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:26.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:26.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:26.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:33:26.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:26.656115+0000) 2019-10-18T02:33:26.703 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:33:26.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:26.703396+0000) 2019-10-18T02:33:26.801 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:33:26.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:26.799954+0000) 2019-10-18T02:33:26.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:26.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:26.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:26.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:27.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:27.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:27.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:27.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:28.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:28.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:28.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:28.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:29.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:29.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:29.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:29.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:30.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:30.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:30.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:30.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:31.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:31.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:31.793 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:33:31.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:31.792875+0000) 2019-10-18T02:33:31.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:31.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:31.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:31.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:32.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:32.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:32.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:32.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:33.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:33.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:33.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:33.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:34.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:34.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:34.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:34.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:35.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:35.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:35.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:35.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:36.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:36.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:36.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:33:36.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:36.656219+0000) 2019-10-18T02:33:36.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:33:36.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:36.703496+0000) 2019-10-18T02:33:36.800 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:33:36.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:36.800161+0000) 2019-10-18T02:33:36.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:36.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:36.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:36.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:37.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:37.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:37.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:37.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:38.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:38.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:38.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:38.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:39.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:39.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:39.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:39.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:40.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:40.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:40.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:40.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:41.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:41.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:41.793 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:33:41.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:41.793103+0000) 2019-10-18T02:33:41.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:41.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:41.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:41.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:42.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:42.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:42.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:42.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:43.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:43.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:43.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:43.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:44.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:44.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:44.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:44.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:46.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:46.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:46.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:46.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:46.186 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:46.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:46.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:33:46.650+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:46.656317+0000) 2019-10-18T02:33:46.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:33:46.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:46.703604+0000) 2019-10-18T02:33:46.801 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:33:46.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:46.800366+0000) 2019-10-18T02:33:47.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:47.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:47.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:47.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:47.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:47.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:47.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:47.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:49.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:49.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:49.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:49.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:49.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:49.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:49.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:49.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:51.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:51.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:51.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:51.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:51.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:51.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:51.794 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:33:51.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:51.793344+0000) 2019-10-18T02:33:51.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:51.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:51.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:51.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:52.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:52.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:52.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:52.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:53.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:53.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:53.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:53.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:55.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:55.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:55.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:55.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:56.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:56.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:56.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:56.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:56.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:33:56.181+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:33:56.661 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:33:56.649+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:56.656419+0000) 2019-10-18T02:33:56.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:33:56.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:56.703711+0000) 2019-10-18T02:33:56.801 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:33:56.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:33:56.800568+0000) 2019-10-18T02:33:57.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:57.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:57.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:57.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:58.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:58.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:58.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:58.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:33:59.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:59.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:33:59.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:33:59.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:00.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:00.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:00.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:00.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:01.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:01.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:01.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:01.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:01.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:01.181+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:01.794 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:34:01.789+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:01.793563+0000) 2019-10-18T02:34:02.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:02.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:02.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:02.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:03.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:03.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:03.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:03.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:04.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:04.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:04.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:04.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:05.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:05.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:05.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:05.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:06.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:06.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:06.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:06.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:06.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:06.181+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:06.656 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:34:06.649+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:06.656529+0000) 2019-10-18T02:34:06.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:34:06.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:06.703808+0000) 2019-10-18T02:34:06.801 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:06.797+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:06.800768+0000) 2019-10-18T02:34:06.802 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:06.797+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:06.801851+0000) 2019-10-18T02:34:07.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:07.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:07.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:07.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:08.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:08.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:08.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:08.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:09.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:09.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:09.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:09.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:10.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:10.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:10.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:10.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:11.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:11.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:11.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:11.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:11.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:11.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:11.301 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:11.296+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:11.301287+0000) 2019-10-18T02:34:11.794 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:34:11.788+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:11.793827+0000) 2019-10-18T02:34:12.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:12.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:12.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:12.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:13.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:13.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:13.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:13.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:14.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:14.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:14.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:14.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:15.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:15.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:15.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:15.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:16.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:16.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:16.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:16.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:16.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:16.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:16.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:34:16.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:16.656626+0000) 2019-10-18T02:34:16.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:34:16.701+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:16.703913+0000) 2019-10-18T02:34:17.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:17.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:17.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:17.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:17.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:17.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:17.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:17.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:18.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:18.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:18.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:18.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:19.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:19.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:19.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:19.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:20.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:20.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:20.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:20.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:21.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:21.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:21.301 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:21.296+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:21.301449+0000) 2019-10-18T02:34:21.794 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:34:21.788+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:21.794078+0000) 2019-10-18T02:34:21.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:21.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:21.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:21.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:22.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:22.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:22.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:22.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:23.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:23.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:23.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:23.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:24.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:24.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:24.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:24.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:25.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:25.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:25.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:25.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:26.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:26.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:26.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:34:26.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:26.656731+0000) 2019-10-18T02:34:26.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:34:26.701+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:26.704020+0000) 2019-10-18T02:34:26.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:26.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:26.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:26.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:27.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:27.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:27.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:27.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:28.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:28.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:28.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:28.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:29.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:29.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:29.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:29.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:30.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:30.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:30.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:30.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:31.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:31.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:31.302 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:31.296+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:31.301581+0000) 2019-10-18T02:34:31.794 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:34:31.792+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:31.794335+0000) 2019-10-18T02:34:31.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:31.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:31.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:31.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:32.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:32.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:32.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:32.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:33.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:33.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:33.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:33.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:34.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:34.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:34.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:34.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:35.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:35.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:35.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:35.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:36.198 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:36.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:36.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:34:36.652+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:36.656835+0000) 2019-10-18T02:34:36.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:34:36.700+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:36.704119+0000) 2019-10-18T02:34:36.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:36.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:36.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:36.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:37.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:37.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:37.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:37.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:38.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:38.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:38.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:38.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:39.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:39.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:39.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:39.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:40.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:40.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:40.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:40.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:41.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:41.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:41.302 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:41.296+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:41.301751+0000) 2019-10-18T02:34:41.795 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:34:41.792+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:41.794532+0000) 2019-10-18T02:34:41.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:41.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:41.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:41.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:42.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:42.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:42.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:42.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:43.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:43.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:43.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:43.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:44.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:44.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:44.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:44.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:45.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:45.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:45.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:45.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:46.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:46.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:46.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:34:46.652+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:46.656933+0000) 2019-10-18T02:34:46.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:34:46.700+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:46.704222+0000) 2019-10-18T02:34:46.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:46.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:46.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:46.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:47.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:47.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:47.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:47.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:48.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:48.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:48.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:48.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:49.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:49.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:49.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:49.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:50.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:50.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:51.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:51.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:51.307 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:34:51.296+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:51.301966+0000) 2019-10-18T02:34:51.795 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:34:51.792+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:51.794747+0000) 2019-10-18T02:34:51.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:51.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:51.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:51.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:52.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:52.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:52.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:52.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:53.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:53.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:53.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:53.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:54.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:54.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:54.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:54.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:55.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:55.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:56.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:34:56.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:34:56.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:34:56.652+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:56.657034+0000) 2019-10-18T02:34:56.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:34:56.700+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:34:56.704322+0000) 2019-10-18T02:34:56.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:56.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:56.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:56.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:57.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:57.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:57.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:57.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:58.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:58.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:58.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:58.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:34:59.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:59.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:34:59.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:34:59.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:00.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:00.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:00.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:00.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:01.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:01.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:01.302 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:35:01.300+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:01.302213+0000) 2019-10-18T02:35:01.795 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:35:01.792+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:01.795006+0000) 2019-10-18T02:35:01.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:01.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:01.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:01.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:02.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:02.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:02.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:02.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:03.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:03.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:03.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:03.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:04.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:04.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:04.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:04.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:05.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:05.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:05.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:05.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:06.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:06.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:06.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:35:06.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:06.657139+0000) 2019-10-18T02:35:06.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:35:06.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:06.704420+0000) 2019-10-18T02:35:06.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:06.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:06.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:06.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:07.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:07.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:07.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:07.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:08.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:08.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:08.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:08.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:09.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:09.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:09.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:09.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:10.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:10.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:10.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:10.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:11.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:11.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:11.302 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:35:11.300+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:11.302400+0000) 2019-10-18T02:35:11.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:11.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:11.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:11.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:11.795 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:35:11.792+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:11.795245+0000) 2019-10-18T02:35:12.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:12.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:12.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:12.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:13.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:13.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:13.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:13.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:14.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:14.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:14.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:14.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:15.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:15.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:15.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:15.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:16.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:16.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:16.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:35:16.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:16.657235+0000) 2019-10-18T02:35:16.704 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:35:16.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:16.704523+0000) 2019-10-18T02:35:16.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:16.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:16.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:16.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:17.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:17.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:17.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:17.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:18.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:18.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:19.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:19.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:19.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:19.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:20.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:20.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:20.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:20.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:21.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:21.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:21.303 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:35:21.300+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:21.302619+0000) 2019-10-18T02:35:21.796 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:35:21.792+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:21.795523+0000) 2019-10-18T02:35:21.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:21.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:21.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:21.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:22.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:22.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:22.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:22.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:23.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:23.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:23.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:23.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:24.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:24.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:24.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:24.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:25.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:25.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:25.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:25.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:26.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:26.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:26.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:35:26.651+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:26.657337+0000) 2019-10-18T02:35:26.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:35:26.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:26.704626+0000) 2019-10-18T02:35:26.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:26.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:26.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:26.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:27.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:27.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:27.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:27.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:28.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:28.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:28.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:28.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:29.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:29.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:29.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:29.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:30.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:30.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:30.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:30.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:31.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:31.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:31.303 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:35:31.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:31.302771+0000) 2019-10-18T02:35:31.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:31.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:31.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:31.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:31.796 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:35:31.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:31.795771+0000) 2019-10-18T02:35:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:32.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:32.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:33.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:33.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:33.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:33.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:34.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:34.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:34.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:34.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:35.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:35.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:35.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:35.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:36.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:36.211+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:36.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:35:36.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:36.657453+0000) 2019-10-18T02:35:36.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:35:36.699+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:36.704731+0000) 2019-10-18T02:35:36.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:36.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:36.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:36.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:37.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:37.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:37.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:37.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:38.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:38.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:38.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:38.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:39.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:39.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:39.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:39.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:40.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:40.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:40.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:40.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:41.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:41.211+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:41.303 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:35:41.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:41.302953+0000) 2019-10-18T02:35:41.796 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:35:41.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:41.796011+0000) 2019-10-18T02:35:41.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:41.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:41.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:41.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:42.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:42.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:42.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:42.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:43.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:43.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:43.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:43.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:44.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:44.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:44.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:44.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:46.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:45.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:46.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:46.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:46.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:46.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:46.657 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:35:46.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:46.657550+0000) 2019-10-18T02:35:46.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:35:46.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:46.704829+0000) 2019-10-18T02:35:47.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:47.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:47.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:47.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:48.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:48.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:48.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:48.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:49.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:49.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:49.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:49.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:50.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:50.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:50.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:50.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:51.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:51.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:51.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:51.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:51.219 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:51.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:51.303 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:35:51.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:51.303101+0000) 2019-10-18T02:35:51.796 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:35:51.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:51.796227+0000) 2019-10-18T02:35:52.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:52.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:52.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:52.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:53.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:53.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:53.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:53.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:54.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:54.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:54.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:54.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:55.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:54.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:55.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:54.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:55.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:55.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:55.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:55.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:56.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:35:56.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:35:56.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:35:56.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:56.657647+0000) 2019-10-18T02:35:56.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:35:56.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:35:56.704937+0000) 2019-10-18T02:35:57.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:57.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:57.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:57.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:57.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:57.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:57.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:57.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:58.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:58.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:58.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:58.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:35:59.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:59.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:35:59.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:35:59.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:00.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:00.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:00.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:00.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:01.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:01.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:01.303 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:36:01.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:01.303334+0000) 2019-10-18T02:36:01.797 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:36:01.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:01.796445+0000) 2019-10-18T02:36:01.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:01.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:01.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:01.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:02.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:02.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:02.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:02.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:03.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:03.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:03.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:03.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:05.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:04.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:05.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:04.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:06.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:06.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:06.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:06.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:06.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:06.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:06.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:36:06.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:06.657743+0000) 2019-10-18T02:36:06.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:36:06.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:06.705045+0000) 2019-10-18T02:36:07.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:07.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:07.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:07.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:08.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:08.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:08.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:08.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:09.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:09.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:09.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:09.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:10.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:10.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:10.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:10.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:11.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:11.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:11.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:11.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:11.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:11.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:11.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:36:11.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:11.303469+0000) 2019-10-18T02:36:11.797 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:36:11.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:11.796677+0000) 2019-10-18T02:36:12.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:12.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:12.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:12.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:13.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:13.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:13.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:13.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:14.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:14.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:14.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:14.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:15.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:15.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:15.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:15.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:16.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:16.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:16.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:16.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:16.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:16.218+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:16.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:36:16.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:16.657839+0000) 2019-10-18T02:36:16.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:36:16.698+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:16.705148+0000) 2019-10-18T02:36:17.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:17.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:17.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:17.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:18.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:18.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:18.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:18.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:19.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:19.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:19.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:19.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:20.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:20.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:20.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:20.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:21.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:21.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:21.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:21.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:21.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:21.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:21.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:36:21.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:21.303635+0000) 2019-10-18T02:36:21.797 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:36:21.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:21.796892+0000) 2019-10-18T02:36:22.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:22.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:22.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:22.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:23.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:23.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:23.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:23.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:24.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:24.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:24.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:24.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:25.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:25.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:25.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:25.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:26.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:26.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:26.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:26.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:26.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:26.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:26.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:36:26.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:26.657941+0000) 2019-10-18T02:36:26.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:36:26.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:26.705249+0000) 2019-10-18T02:36:27.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:27.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:27.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:27.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:28.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:28.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:28.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:28.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:29.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:29.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:29.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:29.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:30.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:30.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:30.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:30.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:31.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:31.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:31.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:31.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:31.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:31.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:31.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:36:31.299+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:31.303796+0000) 2019-10-18T02:36:31.797 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:36:31.791+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:31.797145+0000) 2019-10-18T02:36:32.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:32.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:32.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:32.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:33.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:33.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:33.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:33.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:34.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:34.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:34.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:34.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:35.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:35.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:35.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:35.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:36.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:36.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:36.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:36.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:36.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:36.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:36.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:36:36.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:36.658054+0000) 2019-10-18T02:36:36.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:36:36.697+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:36.705351+0000) 2019-10-18T02:36:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:37.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:37.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:38.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:38.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:38.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:38.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:38.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:38.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:38.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:38.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:39.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:39.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:39.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:39.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:41.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:41.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:41.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:41.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:41.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:41.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:41.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:36:41.298+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:41.303967+0000) 2019-10-18T02:36:41.800 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:36:41.790+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:41.797401+0000) 2019-10-18T02:36:42.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:42.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:42.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:42.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:43.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:43.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:43.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:43.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:44.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:44.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:44.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:44.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:45.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:45.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:45.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:45.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:46.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:46.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:46.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:46.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:46.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:46.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:46.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:36:46.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:46.658147+0000) 2019-10-18T02:36:46.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:36:46.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:46.705457+0000) 2019-10-18T02:36:47.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:47.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:47.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:47.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:48.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:48.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:48.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:48.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:49.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:49.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:49.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:49.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:50.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:50.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:50.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:51.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:51.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:51.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:51.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:51.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:51.229+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:51.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:36:51.298+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:51.304105+0000) 2019-10-18T02:36:51.798 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:36:51.790+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:51.797646+0000) 2019-10-18T02:36:52.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:52.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:52.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:52.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:53.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:53.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:53.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:53.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:54.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:54.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:54.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:54.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:55.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:55.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:56.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:56.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:56.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:56.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:56.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:36:56.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:36:56.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:36:56.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:56.658246+0000) 2019-10-18T02:36:56.705 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:36:56.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:36:56.705557+0000) 2019-10-18T02:36:57.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:57.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:57.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:57.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:58.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:58.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:58.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:58.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:59.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:58.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:59.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:59.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:36:59.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:59.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:36:59.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:36:59.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:00.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:00.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:00.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:00.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:01.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:01.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:01.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:37:01.298+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:01.304262+0000) 2019-10-18T02:37:01.798 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:37:01.790+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:01.797858+0000) 2019-10-18T02:37:01.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:01.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:01.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:01.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:02.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:02.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:02.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:02.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:03.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:03.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:03.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:03.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:04.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:04.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:04.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:04.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:05.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:05.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:05.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:05.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:06.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:06.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:06.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:37:06.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:06.658344+0000) 2019-10-18T02:37:06.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:37:06.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:06.705662+0000) 2019-10-18T02:37:06.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:06.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:06.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:06.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:08.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:08.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:08.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:08.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:09.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:09.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:09.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:09.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:10.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:10.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:10.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:10.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:11.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:11.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:11.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:11.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:11.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:11.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:11.304 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:37:11.298+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:11.304409+0000) 2019-10-18T02:37:11.798 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:37:11.790+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:11.798079+0000) 2019-10-18T02:37:12.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:12.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:12.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:12.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:13.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:13.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:13.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:13.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:14.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:14.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:14.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:14.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:15.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:15.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:15.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:15.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:15.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:15.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:15.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:15.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:16.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:16.232+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:16.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:37:16.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:16.658444+0000) 2019-10-18T02:37:16.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:37:16.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:16.705765+0000) 2019-10-18T02:37:17.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:17.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:17.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:17.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:18.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:18.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:18.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:18.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:19.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:19.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:19.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:19.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:20.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:20.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:20.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:20.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:21.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:21.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:21.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:21.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:21.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:21.236+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:21.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:37:21.298+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:21.304566+0000) 2019-10-18T02:37:21.798 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:37:21.794+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:21.798312+0000) 2019-10-18T02:37:22.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:22.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:22.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:22.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:23.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:23.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:23.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:23.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:24.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:24.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:24.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:24.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:25.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:25.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:25.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:25.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:26.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:26.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:26.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:26.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:26.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:26.236+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:26.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:37:26.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:26.658543+0000) 2019-10-18T02:37:26.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:37:26.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:26.705862+0000) 2019-10-18T02:37:27.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:27.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:27.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:27.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:28.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:28.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:28.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:28.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:29.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:29.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:29.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:29.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:30.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:30.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:30.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:30.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:31.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:31.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:31.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:31.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:31.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:31.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:31.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:37:31.298+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:31.304730+0000) 2019-10-18T02:37:31.799 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:37:31.794+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:31.798519+0000) 2019-10-18T02:37:32.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:32.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:32.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:32.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:33.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:33.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:33.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:33.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:34.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:34.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:34.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:34.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:35.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:35.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:35.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:35.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:36.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:36.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:36.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:36.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:36.242 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:36.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:36.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:37:36.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:36.658642+0000) 2019-10-18T02:37:36.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:37:36.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:36.705959+0000) 2019-10-18T02:37:37.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:37.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:37.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:37.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:38.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:38.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:38.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:38.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:39.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:39.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:39.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:39.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:40.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:40.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:40.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:40.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:41.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:40.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:41.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:40.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:41.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:41.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:41.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:37:41.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:41.304902+0000) 2019-10-18T02:37:41.799 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:37:41.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:41.798796+0000) 2019-10-18T02:37:42.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:42.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:42.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:42.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:43.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:43.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:43.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:43.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:44.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:44.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:44.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:44.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:45.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:44.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:45.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:44.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:46.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:46.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:46.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:46.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:46.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:46.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:46.658 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:37:46.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:46.658740+0000) 2019-10-18T02:37:46.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:37:46.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:46.706055+0000) 2019-10-18T02:37:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:46.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:46.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:48.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:48.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:48.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:48.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:49.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:49.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:49.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:49.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:50.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:50.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:50.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:50.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:51.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:51.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:51.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:51.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:51.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:51.243+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:51.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:37:51.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:51.305097+0000) 2019-10-18T02:37:51.799 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:37:51.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:51.799032+0000) 2019-10-18T02:37:52.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:52.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:52.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:52.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:53.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:53.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:53.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:53.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:54.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:54.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:54.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:54.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:55.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:55.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:55.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:56.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:56.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:56.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:56.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:56.247 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:37:56.243+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:37:56.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:37:56.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:56.658843+0000) 2019-10-18T02:37:56.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:37:56.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:37:56.706154+0000) 2019-10-18T02:37:57.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:57.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:57.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:57.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:58.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:58.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:58.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:58.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:37:59.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:59.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:37:59.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:37:59.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:00.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:00.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:01.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:01.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:01.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:01.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:01.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:01.243+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:01.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:38:01.301+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:01.305269+0000) 2019-10-18T02:38:01.799 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:38:01.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:01.799271+0000) 2019-10-18T02:38:02.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:02.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:02.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:02.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:03.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:03.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:03.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:03.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:04.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:04.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:04.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:04.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:05.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:05.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:05.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:05.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:06.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:06.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:06.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:06.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:06.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:06.246+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:06.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:38:06.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:06.658945+0000) 2019-10-18T02:38:06.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:38:06.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:06.706255+0000) 2019-10-18T02:38:07.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:07.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:07.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:07.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:08.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:08.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:08.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:08.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:09.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:09.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:09.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:09.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:10.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:10.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:10.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:10.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:11.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:11.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:11.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:11.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:11.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:11.246+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:11.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:38:11.301+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:11.305427+0000) 2019-10-18T02:38:11.800 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:38:11.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:11.799540+0000) 2019-10-18T02:38:12.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:12.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:12.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:12.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:13.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:13.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:13.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:13.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:14.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:14.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:14.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:14.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:15.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:15.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:15.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:15.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:16.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:16.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:16.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:16.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:16.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:16.246+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:16.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:38:16.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:16.659041+0000) 2019-10-18T02:38:16.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:38:16.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:16.706364+0000) 2019-10-18T02:38:17.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:17.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:17.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:17.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:18.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:18.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:18.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:18.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:19.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:19.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:19.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:19.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:20.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:20.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:20.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:20.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:21.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:21.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:21.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:21.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:21.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:21.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:21.305 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:38:21.301+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:21.305598+0000) 2019-10-18T02:38:21.800 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:38:21.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:21.799769+0000) 2019-10-18T02:38:22.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:22.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:22.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:22.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:23.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:23.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:23.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:23.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:24.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:24.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:24.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:24.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:25.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:25.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:25.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:25.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:26.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:26.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:26.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:26.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:26.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:26.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:26.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:38:26.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:26.659145+0000) 2019-10-18T02:38:26.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:38:26.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:26.706464+0000) 2019-10-18T02:38:27.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:27.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:27.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:27.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:28.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:28.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:28.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:28.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:29.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:29.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:29.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:29.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:30.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:30.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:30.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:30.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:31.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:31.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:31.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:31.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:31.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:31.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:31.306 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:38:31.301+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:31.305770+0000) 2019-10-18T02:38:31.800 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:38:31.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:31.799991+0000) 2019-10-18T02:38:32.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:32.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:32.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:32.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:33.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:33.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:33.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:33.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:34.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:34.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:34.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:34.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:35.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:35.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:35.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:35.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:36.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:36.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:36.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:36.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:36.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:36.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:36.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:38:36.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:36.659251+0000) 2019-10-18T02:38:36.706 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:38:36.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:36.706586+0000) 2019-10-18T02:38:37.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:37.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:37.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:37.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:38.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:38.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:38.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:38.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:39.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:39.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:39.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:39.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:40.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:40.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:40.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:40.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:41.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:41.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:41.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:41.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:41.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:41.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:41.307 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:38:41.301+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:41.305940+0000) 2019-10-18T02:38:41.800 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:38:41.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:41.800240+0000) 2019-10-18T02:38:42.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:42.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:42.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:42.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:43.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:43.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:43.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:43.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:44.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:44.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:44.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:44.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:45.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:45.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:45.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:45.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:46.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:46.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:46.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:46.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:46.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:46.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:38:46.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:46.659351+0000) 2019-10-18T02:38:46.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:38:46.701+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:46.706685+0000) 2019-10-18T02:38:47.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:47.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:47.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:47.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:48.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:48.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:48.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:48.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:49.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:49.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:49.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:49.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:50.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:50.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:50.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:50.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:51.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:51.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:51.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:51.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:51.263 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:51.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:51.306 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:38:51.301+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:51.306137+0000) 2019-10-18T02:38:51.800 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:38:51.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:51.800496+0000) 2019-10-18T02:38:52.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:52.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:52.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:52.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:53.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:53.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:53.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:53.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:54.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:54.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:54.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:54.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:55.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:55.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:55.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:55.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:56.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:56.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:56.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:56.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:56.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:38:56.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:38:56.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:38:56.653+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:56.659453+0000) 2019-10-18T02:38:56.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:38:56.701+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:38:56.706789+0000) 2019-10-18T02:38:57.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:57.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:57.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:57.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:58.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:58.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:58.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:58.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:38:59.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:59.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:38:59.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:38:59.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:00.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:00.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:00.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:00.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:01.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:01.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:01.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:01.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:01.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:01.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:01.306 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:39:01.305+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:01.306411+0000) 2019-10-18T02:39:01.801 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:39:01.797+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:01.800741+0000) 2019-10-18T02:39:02.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:02.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:02.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:02.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:03.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:03.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:03.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:03.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:04.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:04.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:04.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:04.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:05.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:05.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:05.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:06.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:06.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:06.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:06.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:06.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:06.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:06.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:39:06.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:06.659558+0000) 2019-10-18T02:39:06.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:39:06.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:06.706890+0000) 2019-10-18T02:39:07.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:07.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:07.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:07.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:08.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:08.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:08.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:08.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:09.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:09.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:09.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:09.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:10.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:10.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:10.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:10.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:11.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:11.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:11.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:11.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:11.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:11.265+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:11.306 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:39:11.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:11.306581+0000) 2019-10-18T02:39:11.801 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:39:11.796+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:11.800957+0000) 2019-10-18T02:39:12.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:12.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:12.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:12.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:13.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:13.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:13.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:13.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:14.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:14.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:14.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:14.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:15.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:15.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:15.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:15.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:16.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:16.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:16.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:16.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:16.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:16.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:16.659 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:39:16.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:16.659654+0000) 2019-10-18T02:39:16.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:39:16.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:16.706996+0000) 2019-10-18T02:39:17.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:17.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:17.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:17.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:18.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:18.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:18.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:18.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:19.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:19.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:19.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:19.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:20.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:20.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:20.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:20.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:21.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:21.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:21.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:21.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:21.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:21.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:21.307 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:39:21.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:21.306757+0000) 2019-10-18T02:39:21.802 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:39:21.796+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:21.801186+0000) 2019-10-18T02:39:22.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:22.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:22.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:22.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:23.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:23.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:23.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:23.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:24.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:24.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:24.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:24.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:25.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:25.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:25.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:25.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:26.269 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:26.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:26.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:26.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:28:53.606759+0000 (oldest deadline 2019-10-18T02:29:13.606759+0000) 2019-10-18T02:39:26.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:26.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:26.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:39:26.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:26.659761+0000) 2019-10-18T02:39:26.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:39:26.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:26.707102+0000) 2019-10-18T02:39:27.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:27.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:28.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:28.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:29.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:29.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:30.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:30.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:31.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:31.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:31.307 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:39:31.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:31.306945+0000) 2019-10-18T02:39:31.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:31.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:31.801 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:39:31.796+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:31.801400+0000) 2019-10-18T02:39:32.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:32.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:33.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:33.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:34.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:34.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:35.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:35.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:36.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:36.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:36.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:36.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:36.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:39:36.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:36.659874+0000) 2019-10-18T02:39:36.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:39:36.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:36.707203+0000) 2019-10-18T02:39:37.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:37.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:38.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:38.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:39.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:39.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:40.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:40.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:41.271 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:41.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:41.307 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:39:41.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:41.307160+0000) 2019-10-18T02:39:41.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:41.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:41.802 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:39:41.796+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:41.801643+0000) 2019-10-18T02:39:42.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:42.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:43.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:43.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:44.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:44.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:45.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:45.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:46.274 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:46.272+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:46.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:46.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:46.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:39:46.656+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:46.659981+0000) 2019-10-18T02:39:46.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:39:46.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:46.707304+0000) 2019-10-18T02:39:47.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:47.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:48.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:48.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:49.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:49.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:50.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:50.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:50.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:50.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:51.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:51.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:51.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:51.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:51.278 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:51.276+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:51.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:39:51.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:51.307352+0000) 2019-10-18T02:39:51.802 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:39:51.796+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:51.801876+0000) 2019-10-18T02:39:52.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:52.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:52.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:52.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:53.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:53.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:53.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:53.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:54.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:54.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:54.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:54.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:55.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:55.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:55.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:55.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:56.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:39:56.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:39:56.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:56.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:56.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:56.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:56.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:39:56.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:56.660082+0000) 2019-10-18T02:39:56.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:39:56.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:39:56.707403+0000) 2019-10-18T02:39:57.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:57.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:57.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:57.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:58.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:58.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:58.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:58.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:39:59.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:59.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:39:59.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:39:59.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:00.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:00.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:00.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:00.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:01.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:01.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:01.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:01.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:01.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:01.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:01.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:40:01.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:01.307565+0000) 2019-10-18T02:40:01.804 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:40:01.796+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:01.802120+0000) 2019-10-18T02:40:02.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:02.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:02.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:02.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:03.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:03.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:03.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:03.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:04.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:04.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:04.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:04.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:05.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:05.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:05.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:05.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:06.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:06.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:06.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:06.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:06.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:06.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:06.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:40:06.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:06.660177+0000) 2019-10-18T02:40:06.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:40:06.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:06.707508+0000) 2019-10-18T02:40:07.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:07.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:07.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:07.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:08.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:08.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:08.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:08.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:09.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:09.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:09.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:09.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:10.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:10.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:10.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:10.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:11.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:11.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:11.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:11.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:11.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:11.279+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:11.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:40:11.304+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:11.307746+0000) 2019-10-18T02:40:11.802 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:40:11.800+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:11.802368+0000) 2019-10-18T02:40:12.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:12.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:12.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:12.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:13.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:13.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:13.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:13.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:14.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:14.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:14.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:14.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:15.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:15.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:15.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:15.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:16.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:16.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:16.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:16.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:16.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:16.279+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:16.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:40:16.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:16.660274+0000) 2019-10-18T02:40:16.707 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:40:16.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:16.707612+0000) 2019-10-18T02:40:17.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:17.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:17.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:17.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:18.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:18.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:18.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:18.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:19.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:19.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:19.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:19.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:20.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:20.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:20.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:20.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:21.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:21.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:21.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:21.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:21.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:21.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:21.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:40:21.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:21.307980+0000) 2019-10-18T02:40:21.804 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:40:21.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:21.802588+0000) 2019-10-18T02:40:22.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:22.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:22.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:22.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:23.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:23.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:23.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:23.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:24.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:24.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:24.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:24.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:25.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:25.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:25.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:25.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:26.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:26.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:26.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:26.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:26.286 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:26.283+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:26.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:40:26.655+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:26.660375+0000) 2019-10-18T02:40:26.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:40:26.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:26.707716+0000) 2019-10-18T02:40:27.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:27.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:27.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:27.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:28.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:28.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:28.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:28.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:29.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:29.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:29.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:29.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:30.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:30.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:30.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:30.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:31.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:31.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:31.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:31.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:31.287 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:31.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:31.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:40:31.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:31.308173+0000) 2019-10-18T02:40:31.803 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:40:31.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:31.802807+0000) 2019-10-18T02:40:32.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:32.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:32.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:32.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:33.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:33.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:33.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:33.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:34.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:34.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:34.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:34.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:35.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:35.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:35.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:35.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:36.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:36.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:36.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:36.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:36.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:36.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:36.661 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:40:36.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:36.660479+0000) 2019-10-18T02:40:36.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:40:36.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:36.707821+0000) 2019-10-18T02:40:37.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:37.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:37.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:37.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:38.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:38.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:39.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:39.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:39.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:39.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:40.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:40.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:40.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:40.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:41.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:41.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:41.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:41.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:41.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:41.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:41.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:40:41.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:41.308343+0000) 2019-10-18T02:40:41.803 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:40:41.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:41.802982+0000) 2019-10-18T02:40:42.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:42.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:42.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:42.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:43.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:43.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:43.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:43.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:44.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:44.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:44.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:44.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:45.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:45.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:45.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:45.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:46.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:46.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:46.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:46.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:46.290 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:46.286+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:46.660 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:40:46.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:46.660577+0000) 2019-10-18T02:40:46.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:40:46.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:46.707926+0000) 2019-10-18T02:40:47.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:47.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:47.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:47.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:48.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:48.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:48.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:48.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:49.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:49.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:49.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:49.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:50.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:50.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:50.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:50.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:51.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:51.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:51.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:51.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:51.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:51.286+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:51.308 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:40:51.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:51.308498+0000) 2019-10-18T02:40:51.803 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:40:51.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:51.803174+0000) 2019-10-18T02:40:52.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:52.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:52.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:52.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:53.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:53.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:53.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:53.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:54.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:54.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:54.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:54.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:55.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:55.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:55.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:55.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:56.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:56.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:56.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:56.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:56.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:40:56.286+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:40:56.661 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:40:56.654+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:56.660686+0000) 2019-10-18T02:40:56.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:40:56.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:40:56.708025+0000) 2019-10-18T02:40:57.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:57.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:57.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:57.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:58.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:58.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:58.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:58.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:40:59.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:59.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:40:59.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:40:59.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:00.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:00.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:00.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:00.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:01.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:01.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:01.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:01.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:01.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:01.286+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:01.309 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:41:01.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:01.308683+0000) 2019-10-18T02:41:01.803 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:41:01.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:01.803390+0000) 2019-10-18T02:41:02.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:02.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:02.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:02.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:03.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:03.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:03.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:03.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:04.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:04.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:04.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:04.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:05.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:05.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:05.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:05.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:06.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:06.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:06.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:06.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:06.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:06.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:06.661 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:41:06.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:06.661493+0000) 2019-10-18T02:41:06.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:41:06.701+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:06.708127+0000) 2019-10-18T02:41:07.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:07.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:07.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:07.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:08.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:08.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:08.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:08.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:09.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:09.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:09.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:09.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:10.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:10.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:10.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:10.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:11.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:11.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:11.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:11.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:11.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:11.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:11.309 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:41:11.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:11.308872+0000) 2019-10-18T02:41:11.803 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:41:11.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:11.803621+0000) 2019-10-18T02:41:12.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:12.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:12.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:12.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:13.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:13.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:13.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:13.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:14.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:14.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:14.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:14.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:15.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:15.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:15.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:15.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:16.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:16.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:16.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:16.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:16.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:16.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:16.661 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:41:16.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:16.661592+0000) 2019-10-18T02:41:16.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:41:16.701+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:16.708225+0000) 2019-10-18T02:41:17.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:17.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:17.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:17.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:18.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:18.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:18.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:18.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:19.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:19.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:19.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:19.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:20.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:20.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:20.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:20.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:21.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:21.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:21.309 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:41:21.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:21.309037+0000) 2019-10-18T02:41:21.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:21.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:21.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:21.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:21.804 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:41:21.799+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:21.803840+0000) 2019-10-18T02:41:22.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:22.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:22.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:22.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:23.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:23.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:23.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:23.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:24.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:24.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:24.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:24.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:25.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:25.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:25.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:25.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:26.297 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:26.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:26.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:26.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:26.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:26.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:26.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:41:26.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:26.661687+0000) 2019-10-18T02:41:26.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:41:26.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:26.708324+0000) 2019-10-18T02:41:27.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:27.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:27.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:27.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:28.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:28.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:28.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:28.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:29.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:29.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:29.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:29.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:30.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:30.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:30.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:30.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:31.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:31.297+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:31.309 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:41:31.303+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:31.309232+0000) 2019-10-18T02:41:31.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:31.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:31.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:31.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:31.804 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:41:31.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:31.804065+0000) 2019-10-18T02:41:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:32.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:32.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:32.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:33.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:33.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:33.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:33.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:34.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:34.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:34.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:34.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:35.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:35.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:35.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:35.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:36.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:36.297+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:36.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:36.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:36.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:36.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:36.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:41:36.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:36.661785+0000) 2019-10-18T02:41:36.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:41:36.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:36.708424+0000) 2019-10-18T02:41:37.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:37.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:37.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:38.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:38.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:38.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:38.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:39.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:39.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:39.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:39.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:40.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:40.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:40.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:40.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:41.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:41.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:41.309 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:41:41.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:41.309401+0000) 2019-10-18T02:41:41.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:41.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:41.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:41.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:41.804 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:41:41.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:41.804321+0000) 2019-10-18T02:41:42.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:42.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:42.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:42.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:43.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:43.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:43.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:43.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:44.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:44.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:44.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:44.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:45.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:45.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:45.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:45.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:46.301 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:46.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:46.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:46.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:46.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:46.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:46.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:41:46.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:46.661889+0000) 2019-10-18T02:41:46.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:41:46.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:46.708529+0000) 2019-10-18T02:41:47.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:47.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:47.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:47.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:48.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:48.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:48.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:48.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:49.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:49.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:49.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:49.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:50.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:50.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:50.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:50.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:51.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:51.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:51.309 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:41:51.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:51.309588+0000) 2019-10-18T02:41:51.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:51.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:51.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:51.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:51.804 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:41:51.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:51.804571+0000) 2019-10-18T02:41:52.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:52.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:52.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:52.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:53.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:53.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:53.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:54.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:54.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:54.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:54.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:55.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:55.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:55.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:55.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:56.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:41:56.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:41:56.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:56.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:56.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:56.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:56.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:41:56.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:56.661999+0000) 2019-10-18T02:41:56.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:41:56.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:41:56.708632+0000) 2019-10-18T02:41:57.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:57.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:57.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:57.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:58.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:58.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:41:59.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:59.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:41:59.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:41:59.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:00.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:00.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:00.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:00.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:01.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:01.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:01.310 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:42:01.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:01.309744+0000) 2019-10-18T02:42:01.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:01.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:01.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:01.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:01.805 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:42:01.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:01.804807+0000) 2019-10-18T02:42:02.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:02.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:02.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:02.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:03.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:03.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:03.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:03.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:04.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:04.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:04.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:04.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:05.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:05.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:05.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:05.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:06.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:06.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:06.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:06.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:06.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:06.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:06.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:42:06.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:06.662111+0000) 2019-10-18T02:42:06.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:42:06.704+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:06.708730+0000) 2019-10-18T02:42:07.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:07.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:07.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:07.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:08.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:08.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:08.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:08.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:09.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:09.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:09.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:09.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:10.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:10.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:10.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:10.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:11.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:11.300+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:11.310 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:42:11.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:11.309894+0000) 2019-10-18T02:42:11.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:11.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:11.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:11.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:11.805 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:42:11.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:11.805052+0000) 2019-10-18T02:42:12.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:12.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:12.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:12.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:13.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:13.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:13.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:13.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:14.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:14.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:14.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:14.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:15.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:15.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:15.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:16.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:16.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:16.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:16.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:16.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:16.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:42:16.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:16.662216+0000) 2019-10-18T02:42:16.708 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:42:16.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:16.708834+0000) 2019-10-18T02:42:17.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:17.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:17.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:17.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:18.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:18.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:18.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:18.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:19.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:19.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:19.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:19.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:20.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:20.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:20.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:20.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:21.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:21.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:21.310 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:42:21.302+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:21.310120+0000) 2019-10-18T02:42:21.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:21.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:21.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:21.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:21.805 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:42:21.798+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:21.805305+0000) 2019-10-18T02:42:22.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:22.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:22.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:22.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:23.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:23.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:23.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:23.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:24.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:24.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:24.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:24.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:25.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:25.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:25.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:25.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:26.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:26.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:26.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:26.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:26.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:26.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:26.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:42:26.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:26.662315+0000) 2019-10-18T02:42:26.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:42:26.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:26.708937+0000) 2019-10-18T02:42:27.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:27.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:27.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:27.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:28.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:28.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:28.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:28.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:29.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:29.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:29.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:29.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:30.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:30.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:30.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:30.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:31.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:31.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:31.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:31.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:31.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:31.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:31.310 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:42:31.306+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:31.310318+0000) 2019-10-18T02:42:31.806 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:42:31.802+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:31.805571+0000) 2019-10-18T02:42:32.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:32.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:32.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:32.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:33.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:33.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:33.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:33.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:34.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:34.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:34.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:34.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:35.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:35.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:35.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:35.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:36.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:36.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:36.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:36.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:36.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:36.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:36.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:42:36.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:36.662418+0000) 2019-10-18T02:42:36.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:42:36.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:36.709033+0000) 2019-10-18T02:42:37.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:37.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:37.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:37.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:38.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:38.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:39.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:39.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:39.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:39.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:40.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:40.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:40.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:40.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:41.311 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:42:41.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:41.310476+0000) 2019-10-18T02:42:41.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:41.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:41.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:41.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:41.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:41.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:41.806 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:42:41.802+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:41.805819+0000) 2019-10-18T02:42:42.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:42.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:42.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:42.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:43.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:43.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:43.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:43.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:44.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:44.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:44.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:44.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:45.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:45.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:45.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:45.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:46.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:46.307+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:46.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:46.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:46.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:46.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:46.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:42:46.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:46.662519+0000) 2019-10-18T02:42:46.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:42:46.703+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:46.709130+0000) 2019-10-18T02:42:47.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:47.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:47.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:47.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:48.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:48.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:48.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:48.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:49.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:49.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:49.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:49.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:50.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:50.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:50.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:50.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:51.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:51.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:51.313 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:42:51.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:51.310691+0000) 2019-10-18T02:42:51.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:51.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:51.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:51.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:51.806 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:42:51.801+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:51.806045+0000) 2019-10-18T02:42:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:52.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:52.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:53.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:53.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:53.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:53.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:54.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:54.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:54.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:54.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:55.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:55.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:55.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:55.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:56.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:42:56.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:42:56.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:56.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:56.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:56.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:56.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:42:56.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:56.662615+0000) 2019-10-18T02:42:56.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:42:56.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:42:56.709230+0000) 2019-10-18T02:42:57.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:57.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:57.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:57.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:58.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:58.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:58.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:58.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:42:59.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:59.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:42:59.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:42:59.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:00.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:00.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:00.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:00.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:01.311 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:43:01.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:01.310905+0000) 2019-10-18T02:43:01.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:01.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:01.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:01.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:01.806 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:43:01.805+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:01.806329+0000) 2019-10-18T02:43:02.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:02.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:02.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:02.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:03.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:03.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:03.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:03.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:04.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:04.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:04.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:04.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:05.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:05.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:05.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:05.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:06.314 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:06.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:06.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:06.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:06.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:06.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:06.662 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:43:06.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:06.662719+0000) 2019-10-18T02:43:06.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:43:06.702+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:06.709329+0000) 2019-10-18T02:43:07.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:07.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:07.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:07.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:08.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:08.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:08.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:08.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:09.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:09.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:09.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:09.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:10.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:10.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:10.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:10.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:11.311 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:43:11.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:11.311097+0000) 2019-10-18T02:43:11.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:11.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:11.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:11.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:11.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:11.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:11.806 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:43:11.805+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:11.806524+0000) 2019-10-18T02:43:12.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:12.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:12.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:12.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:13.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:13.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:13.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:13.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:14.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:14.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:14.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:14.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:15.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:15.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:15.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:15.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:16.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:16.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:16.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:16.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:16.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:16.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:16.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:43:16.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:16.662821+0000) 2019-10-18T02:43:16.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:43:16.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:16.709451+0000) 2019-10-18T02:43:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:17.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:17.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:18.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:18.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:18.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:18.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:19.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:19.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:19.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:19.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:20.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:20.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:20.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:20.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:21.311 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:43:21.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:21.311313+0000) 2019-10-18T02:43:21.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:21.310+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:21.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:21.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:21.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:21.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:21.807 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:43:21.805+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:21.806756+0000) 2019-10-18T02:43:22.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:22.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:22.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:22.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:23.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:23.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:23.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:23.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:24.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:24.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:24.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:24.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:25.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:25.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:25.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:25.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:26.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:26.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:26.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:26.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:26.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:26.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:26.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:43:26.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:26.662925+0000) 2019-10-18T02:43:26.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:43:26.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:26.709552+0000) 2019-10-18T02:43:27.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:27.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:27.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:27.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:28.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:28.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:28.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:28.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:29.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:29.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:29.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:29.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:30.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:30.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:30.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:30.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:31.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:43:31.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:31.311497+0000) 2019-10-18T02:43:31.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:31.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:31.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:31.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:31.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:31.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:31.808 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:43:31.805+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:31.806977+0000) 2019-10-18T02:43:32.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:32.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:32.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:32.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:33.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:33.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:33.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:33.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:34.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:34.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:34.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:34.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:35.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:35.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:35.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:35.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:36.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:36.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:36.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:36.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:36.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:36.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:36.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:43:36.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:36.663029+0000) 2019-10-18T02:43:36.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:43:36.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:36.709649+0000) 2019-10-18T02:43:37.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:37.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:37.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:37.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:38.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:38.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:38.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:38.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:39.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:39.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:39.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:39.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:40.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:40.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:40.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:40.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:41.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:43:41.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:41.311682+0000) 2019-10-18T02:43:41.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:41.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:41.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:41.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:41.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:41.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:41.807 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:43:41.805+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:41.807223+0000) 2019-10-18T02:43:42.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:42.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:42.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:42.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:43.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:43.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:43.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:43.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:44.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:44.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:44.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:44.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:45.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:45.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:45.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:45.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:46.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:46.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:46.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:46.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:46.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:46.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:46.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:43:46.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:46.663137+0000) 2019-10-18T02:43:46.709 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:43:46.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:46.709750+0000) 2019-10-18T02:43:47.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:47.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:47.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:47.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:48.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:48.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:48.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:48.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:49.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:49.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:49.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:49.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:50.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:50.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:50.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:50.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:51.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:43:51.309+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:51.311864+0000) 2019-10-18T02:43:51.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:51.321+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:51.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:51.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:51.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:51.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:51.808 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:43:51.805+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:51.807413+0000) 2019-10-18T02:43:52.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:52.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:52.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:52.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:53.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:53.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:53.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:53.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:54.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:54.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:54.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:54.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:55.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:55.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:55.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:55.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:56.323 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:43:56.321+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:43:56.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:56.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:56.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:56.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:56.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:43:56.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:56.663238+0000) 2019-10-18T02:43:56.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:43:56.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:43:56.709848+0000) 2019-10-18T02:43:57.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:57.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:57.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:57.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:58.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:58.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:58.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:58.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:43:59.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:59.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:43:59.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:43:59.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:00.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:00.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:00.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:00.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:01.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:44:01.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:01.312055+0000) 2019-10-18T02:44:01.323 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:01.321+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:01.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:01.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:01.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:01.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:01.807 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:44:01.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:01.807657+0000) 2019-10-18T02:44:02.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:02.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:02.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:02.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:03.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:03.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:03.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:03.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:04.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:04.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:04.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:04.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:05.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:05.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:05.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:05.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:06.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:06.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:06.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:06.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:06.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:06.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:06.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:44:06.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:06.663339+0000) 2019-10-18T02:44:06.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:44:06.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:06.709945+0000) 2019-10-18T02:44:07.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:07.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:07.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:07.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:08.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:08.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:08.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:08.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:09.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:09.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:09.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:09.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:10.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:10.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:10.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:10.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:11.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:44:11.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:11.312245+0000) 2019-10-18T02:44:11.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:11.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:11.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:11.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:11.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:11.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:11.809 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:44:11.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:11.807882+0000) 2019-10-18T02:44:12.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:12.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:12.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:12.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:13.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:13.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:13.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:13.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:14.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:14.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:14.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:14.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:15.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:15.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:15.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:15.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:16.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:16.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:16.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:16.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:16.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:16.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:16.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:44:16.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:16.663448+0000) 2019-10-18T02:44:16.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:44:16.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:16.710050+0000) 2019-10-18T02:44:17.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:17.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:17.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:17.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:18.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:18.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:18.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:18.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:19.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:19.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:19.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:19.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:20.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:20.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:20.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:20.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:21.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:44:21.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:21.312437+0000) 2019-10-18T02:44:21.327 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:21.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:21.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:21.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:21.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:21.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:21.808 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:44:21.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:21.808103+0000) 2019-10-18T02:44:22.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:22.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:22.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:22.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:23.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:23.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:23.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:23.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:24.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:24.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:24.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:24.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:25.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:25.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:25.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:25.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:26.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:26.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:26.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:26.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:26.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:26.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:26.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:44:26.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:26.663546+0000) 2019-10-18T02:44:26.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:44:26.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:26.710146+0000) 2019-10-18T02:44:27.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:27.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:27.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:27.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:28.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:28.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:28.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:28.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:29.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:29.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:29.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:29.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:30.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:30.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:30.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:30.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:31.312 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:44:31.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:31.312628+0000) 2019-10-18T02:44:31.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:31.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:31.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:31.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:31.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:31.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:31.808 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:44:31.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:31.808356+0000) 2019-10-18T02:44:32.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:32.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:32.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:32.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:33.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:33.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:33.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:33.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:34.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:34.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:34.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:34.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:35.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:35.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:35.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:35.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:36.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:36.328+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:36.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:36.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:36.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:36.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:36.663 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:44:36.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:36.663647+0000) 2019-10-18T02:44:36.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:44:36.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:36.710241+0000) 2019-10-18T02:44:37.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:37.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:37.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:37.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:38.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:38.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:38.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:38.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:39.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:39.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:39.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:39.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:40.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:40.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:40.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:40.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:41.313 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:44:41.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:41.312817+0000) 2019-10-18T02:44:41.331 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:41.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:41.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:41.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:41.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:41.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:41.809 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:44:41.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:41.808763+0000) 2019-10-18T02:44:42.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:42.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:42.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:42.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:43.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:43.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:43.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:43.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:44.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:44.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:44.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:44.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:45.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:45.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:45.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:45.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 312 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:46.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:46.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:46.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:46.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:46.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:46.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:46.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:44:46.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:46.663751+0000) 2019-10-18T02:44:46.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:44:46.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:46.710338+0000) 2019-10-18T02:44:47.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:47.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:47.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:47.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:48.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:48.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:48.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:48.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:49.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:49.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:49.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:49.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:50.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:50.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:50.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:50.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:51.315 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:44:51.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:51.313005+0000) 2019-10-18T02:44:51.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:51.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:51.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:51.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:51.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:51.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:51.811 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:44:51.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:51.808978+0000) 2019-10-18T02:44:52.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:52.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:52.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:52.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:53.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:53.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:53.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:53.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:54.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:54.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:54.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:54.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:55.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:55.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:55.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:55.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:56.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:44:56.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:44:56.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:56.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:56.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:56.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:56.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:44:56.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:56.663853+0000) 2019-10-18T02:44:56.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:44:56.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:44:56.710437+0000) 2019-10-18T02:44:57.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:57.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:57.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:57.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:58.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:58.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:58.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:58.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:44:59.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:59.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:44:59.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:44:59.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:00.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:00.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:00.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:00.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:01.313 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:45:01.308+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:01.313191+0000) 2019-10-18T02:45:01.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:01.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:01.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:01.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:01.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:01.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:01.809 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:45:01.804+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:01.809158+0000) 2019-10-18T02:45:02.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:02.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:02.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:02.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:03.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:03.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:03.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:03.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:04.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:04.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:04.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:04.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:05.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:05.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:05.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:05.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:06.335 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:06.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:06.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:06.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:06.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:06.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:06.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:45:06.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:06.663959+0000) 2019-10-18T02:45:06.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:45:06.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:06.710536+0000) 2019-10-18T02:45:07.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:07.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:07.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:07.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:08.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:08.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:08.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:08.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:09.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:09.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:09.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:09.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:10.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:10.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:10.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:10.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:11.313 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:45:11.307+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:11.313338+0000) 2019-10-18T02:45:11.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:11.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:11.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:11.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:11.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:11.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:11.809 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:45:11.803+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:11.809351+0000) 2019-10-18T02:45:12.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:12.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:12.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:12.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:13.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:13.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:13.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:13.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:14.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:14.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:14.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:14.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:15.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:15.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:15.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:15.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:16.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:16.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:16.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:16.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:16.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:16.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:16.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:45:16.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:16.664065+0000) 2019-10-18T02:45:16.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:45:16.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:16.710636+0000) 2019-10-18T02:45:17.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:17.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:17.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:17.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:18.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:18.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:18.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:18.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:19.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:19.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:19.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:19.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:20.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:20.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:20.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:20.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:21.313 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:45:21.307+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:21.313519+0000) 2019-10-18T02:45:21.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:21.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:21.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:21.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:21.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:21.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:21.809 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:45:21.803+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:21.809542+0000) 2019-10-18T02:45:22.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:22.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:22.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:22.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:23.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:23.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:23.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:23.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:24.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:24.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:24.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:24.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:25.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:25.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:25.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:25.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:26.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:26.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:26.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:45:26.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:26.664170+0000) 2019-10-18T02:45:26.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:26.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:26.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:26.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:26.710 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:45:26.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:26.710737+0000) 2019-10-18T02:45:27.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:27.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:27.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:27.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:28.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:28.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:28.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:28.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:29.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:29.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:29.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:29.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:30.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:30.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:30.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:30.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:31.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:45:31.307+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:31.313732+0000) 2019-10-18T02:45:31.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:31.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:31.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:31.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:31.809 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:45:31.803+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:31.809795+0000) 2019-10-18T02:45:32.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:32.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:32.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:32.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:33.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:33.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:33.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:33.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:34.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:34.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:34.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:34.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:35.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:35.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:35.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:35.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:36.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:36.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:36.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:45:36.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:36.664265+0000) 2019-10-18T02:45:36.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:36.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:36.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:36.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:36.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:45:36.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:36.710834+0000) 2019-10-18T02:45:37.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:37.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:37.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:37.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:38.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:38.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:38.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:38.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:39.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:39.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:39.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:39.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:40.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:40.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:40.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:40.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:41.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:45:41.307+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:41.313915+0000) 2019-10-18T02:45:41.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:41.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:41.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:41.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:41.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:41.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:41.810 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:45:41.803+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:41.809999+0000) 2019-10-18T02:45:42.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:42.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:42.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:42.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:43.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:43.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:43.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:43.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:44.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:44.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:45.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:45.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:45.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:45.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:46.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:46.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:46.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:46.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:46.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:46.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:46.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:45:46.658+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:46.664364+0000) 2019-10-18T02:45:46.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:45:46.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:46.710935+0000) 2019-10-18T02:45:47.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:47.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:47.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:47.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:48.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:48.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:48.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:48.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:49.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:49.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:49.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:49.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:50.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:50.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:50.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:50.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:51.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:45:51.307+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:51.314097+0000) 2019-10-18T02:45:51.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:51.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:51.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:51.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:51.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:51.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:51.814 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:45:51.807+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:51.810238+0000) 2019-10-18T02:45:52.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:52.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:52.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:52.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:53.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:53.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:53.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:53.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:54.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:54.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:54.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:54.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:55.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:55.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:55.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:55.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:56.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:45:56.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:45:56.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:45:56.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:56.664472+0000) 2019-10-18T02:45:56.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:56.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:56.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:56.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:56.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:45:56.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:45:56.711035+0000) 2019-10-18T02:45:57.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:57.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:57.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:57.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:58.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:58.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:58.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:58.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:45:59.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:59.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:45:59.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:45:59.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:00.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:00.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:00.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:00.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:01.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:46:01.311+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:01.314307+0000) 2019-10-18T02:46:01.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:01.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:01.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:01.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:01.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:01.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:01.810 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:46:01.807+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:01.810454+0000) 2019-10-18T02:46:02.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:02.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:02.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:02.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:03.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:03.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:03.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:03.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:04.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:04.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:04.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:05.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:05.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:05.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:05.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:06.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:06.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:06.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:46:06.657+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:06.664571+0000) 2019-10-18T02:46:06.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:46:06.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:06.711129+0000) 2019-10-18T02:46:06.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:06.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:06.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:06.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:07.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:07.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:07.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:07.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:08.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:08.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:08.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:08.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:09.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:09.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:09.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:09.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:10.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:10.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:10.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:10.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:11.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:46:11.311+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:11.314484+0000) 2019-10-18T02:46:11.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:11.345+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:11.810 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:46:11.807+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:11.810681+0000) 2019-10-18T02:46:11.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:11.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:11.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:11.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:12.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:12.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:12.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:12.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:13.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:13.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:13.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:13.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:14.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:14.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:14.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:14.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:15.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:15.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:15.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:15.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:16.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:16.345+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:16.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:46:16.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:16.664681+0000) 2019-10-18T02:46:16.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:46:16.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:16.711234+0000) 2019-10-18T02:46:16.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:16.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:16.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:16.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:17.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:17.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:17.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:17.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:18.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:18.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:19.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:19.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:19.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:19.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:20.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:20.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:20.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:20.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:21.314 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:46:21.311+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:21.314699+0000) 2019-10-18T02:46:21.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:21.345+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:21.811 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:46:21.807+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:21.810865+0000) 2019-10-18T02:46:21.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:21.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:21.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:21.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:22.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:22.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:22.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:22.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:23.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:23.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:23.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:23.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:24.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:24.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:24.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:25.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:25.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:25.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:25.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:26.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:26.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:26.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:46:26.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:26.664792+0000) 2019-10-18T02:46:26.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:46:26.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:26.711334+0000) 2019-10-18T02:46:26.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:26.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:26.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:26.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:27.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:27.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:27.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:27.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:28.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:28.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:28.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:28.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:29.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:29.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:29.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:29.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:30.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:30.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:30.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:30.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:31.315 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:46:31.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:31.314910+0000) 2019-10-18T02:46:31.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:31.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:31.811 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:46:31.806+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:31.811075+0000) 2019-10-18T02:46:31.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:31.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:31.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:31.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:32.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:32.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:32.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:32.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:33.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:33.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:33.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:33.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:34.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:34.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:34.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:34.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:35.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:35.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:35.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:35.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:36.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:36.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:36.664 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:46:36.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:36.664907+0000) 2019-10-18T02:46:36.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:46:36.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:36.711428+0000) 2019-10-18T02:46:36.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:36.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:36.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:36.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:37.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:37.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:37.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:37.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:38.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:38.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:38.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:38.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:39.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:39.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:39.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:39.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:40.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:40.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:40.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:40.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:41.315 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:46:41.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:41.315099+0000) 2019-10-18T02:46:41.352 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:41.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:41.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:41.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:41.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:41.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:41.811 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:46:41.806+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:41.811312+0000) 2019-10-18T02:46:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:42.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:42.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:43.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:43.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:43.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:43.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:44.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:44.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:44.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:44.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:45.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:45.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:45.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:45.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:46.352 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:46.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:46.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:46:46.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:46.665012+0000) 2019-10-18T02:46:46.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:46:46.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:46.711523+0000) 2019-10-18T02:46:46.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:46.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:46.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:46.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:47.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:47.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:47.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:47.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:48.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:48.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:48.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:48.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:49.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:49.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:49.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:49.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:50.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:50.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:50.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:50.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:51.315 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:46:51.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:51.315285+0000) 2019-10-18T02:46:51.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:51.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:51.811 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:46:51.806+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:51.811525+0000) 2019-10-18T02:46:51.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:51.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:51.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:51.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:52.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:52.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:52.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:52.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:53.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:53.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:53.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:53.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:54.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:54.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:54.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:54.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:55.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:55.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:56.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:46:56.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:46:56.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:46:56.660+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:56.665114+0000) 2019-10-18T02:46:56.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:46:56.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:46:56.711619+0000) 2019-10-18T02:46:56.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:56.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:56.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:56.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:57.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:57.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:57.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:57.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:58.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:58.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:58.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:58.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:46:59.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:59.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:46:59.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:46:59.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:00.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:00.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:00.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:01.316 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:47:01.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:01.315498+0000) 2019-10-18T02:47:01.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:01.351+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:01.812 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:47:01.806+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:01.811773+0000) 2019-10-18T02:47:01.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:01.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:01.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:01.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:02.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:02.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:02.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:02.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:03.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:03.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:03.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:03.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:04.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:04.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:04.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:04.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:05.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:05.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:05.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:05.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:06.356 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:06.351+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:06.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:47:06.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:06.665219+0000) 2019-10-18T02:47:06.711 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:47:06.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:06.711723+0000) 2019-10-18T02:47:06.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:06.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:06.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:06.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:07.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:07.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:07.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:07.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:08.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:08.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:08.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:08.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:09.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:09.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:09.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:09.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:10.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:10.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:10.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:10.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:11.316 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:47:11.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:11.315708+0000) 2019-10-18T02:47:11.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:11.351+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:11.812 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:47:11.806+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:11.811999+0000) 2019-10-18T02:47:11.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:11.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:11.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:11.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:12.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:12.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:12.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:12.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:13.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:13.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:13.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:13.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:14.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:14.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:14.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:14.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:15.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:15.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:15.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:15.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:16.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:16.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:16.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:47:16.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:16.665327+0000) 2019-10-18T02:47:16.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:47:16.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:16.711825+0000) 2019-10-18T02:47:16.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:16.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:16.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:16.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:17.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:17.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:17.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:17.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:18.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:18.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:19.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:19.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:19.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:19.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:20.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:20.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:20.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:20.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:21.316 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:47:21.310+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:21.315888+0000) 2019-10-18T02:47:21.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:21.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:21.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:21.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:21.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:21.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:21.812 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:47:21.806+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:21.812212+0000) 2019-10-18T02:47:22.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:22.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:22.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:22.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:23.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:23.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:23.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:23.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:24.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:24.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:24.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:24.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:25.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:25.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:25.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:25.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:26.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:26.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:26.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:26.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:26.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:26.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:26.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:47:26.659+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:26.665427+0000) 2019-10-18T02:47:26.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:47:26.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:26.711921+0000) 2019-10-18T02:47:27.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:27.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:27.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:27.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:28.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:28.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:28.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:28.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:29.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:29.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:29.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:29.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:30.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:30.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:30.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:30.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:31.316 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:47:31.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:31.316101+0000) 2019-10-18T02:47:31.360 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:31.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:31.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:31.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:31.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:31.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:31.812 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:47:31.810+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:31.812442+0000) 2019-10-18T02:47:32.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:32.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:32.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:32.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:33.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:33.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:33.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:33.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:34.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:34.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:34.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:34.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:35.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:35.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:35.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:35.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:36.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:36.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:36.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:47:36.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:36.665541+0000) 2019-10-18T02:47:36.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:47:36.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:36.712022+0000) 2019-10-18T02:47:36.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:36.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:36.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:36.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:37.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:37.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:37.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:37.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:38.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:38.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:38.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:38.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:39.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:39.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:39.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:39.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:40.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:40.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:40.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:40.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:41.316 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:47:41.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:41.316291+0000) 2019-10-18T02:47:41.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:41.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:41.812 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:47:41.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:41.812680+0000) 2019-10-18T02:47:41.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:41.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:41.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:41.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:42.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:42.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:42.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:42.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:43.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:43.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:43.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:43.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:44.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:44.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:44.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:44.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:45.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:45.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:45.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:45.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:46.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:46.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:46.665 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:47:46.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:46.665662+0000) 2019-10-18T02:47:46.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:47:46.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:46.712139+0000) 2019-10-18T02:47:46.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:46.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:46.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:46.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:47.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:47.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:47.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:47.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:48.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:48.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:48.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:48.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:49.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:49.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:49.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:49.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:50.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:50.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:50.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:50.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:51.316 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:47:51.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:51.316477+0000) 2019-10-18T02:47:51.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:51.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:51.813 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:47:51.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:51.812917+0000) 2019-10-18T02:47:51.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:51.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:51.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:51.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:52.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:52.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:52.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:52.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:53.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:53.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:53.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:53.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:54.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:54.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:54.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:54.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:55.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:55.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:55.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:55.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:56.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:47:56.358+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:47:56.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:47:56.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:56.665771+0000) 2019-10-18T02:47:56.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:47:56.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:47:56.712250+0000) 2019-10-18T02:47:56.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:56.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:56.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:56.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:57.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:57.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:57.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:57.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:58.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:58.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:58.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:58.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:47:59.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:59.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:47:59.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:47:59.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:00.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:00.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:00.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:00.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:01.317 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:48:01.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:01.316698+0000) 2019-10-18T02:48:01.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:01.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:01.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:01.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:01.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:01.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:01.813 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:48:01.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:01.813127+0000) 2019-10-18T02:48:02.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:02.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:02.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:02.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:03.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:03.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:03.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:03.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:04.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:04.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:04.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:04.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:05.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:05.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:05.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:05.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:06.366 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:06.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:06.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:48:06.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:06.665885+0000) 2019-10-18T02:48:06.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:48:06.706+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:06.712346+0000) 2019-10-18T02:48:06.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:06.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:06.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:06.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:07.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:07.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:07.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:07.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:08.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:08.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:08.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:08.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:09.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:09.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:09.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:09.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:10.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:10.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:10.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:10.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:11.317 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:48:11.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:11.316875+0000) 2019-10-18T02:48:11.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:11.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:11.821 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:48:11.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:11.813347+0000) 2019-10-18T02:48:11.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:11.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:11.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:11.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:12.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:12.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:13.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:13.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:13.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:13.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:14.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:14.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:14.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:14.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:15.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:15.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:15.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:15.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:16.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:16.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:16.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:48:16.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:16.665994+0000) 2019-10-18T02:48:16.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:48:16.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:16.712449+0000) 2019-10-18T02:48:16.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:16.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:16.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:16.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:17.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:17.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:17.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:17.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:18.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:18.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:18.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:18.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:19.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:19.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:19.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:19.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:20.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:20.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:20.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:20.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:21.318 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:48:21.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:21.317061+0000) 2019-10-18T02:48:21.369 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:21.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:21.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:21.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:21.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:21.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:21.814 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:48:21.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:21.813567+0000) 2019-10-18T02:48:22.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:22.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:22.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:22.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:23.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:23.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:23.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:23.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:24.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:24.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:24.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:24.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:25.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:25.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:25.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:25.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:26.370 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:26.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:26.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:26.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:26.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:26.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:26.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:48:26.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:26.666105+0000) 2019-10-18T02:48:26.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:48:26.705+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:26.712550+0000) 2019-10-18T02:48:27.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:27.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:27.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:27.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:28.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:28.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:28.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:28.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:29.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:29.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:29.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:29.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:30.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:30.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:30.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:30.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:31.317 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:48:31.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:31.317272+0000) 2019-10-18T02:48:31.370 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:31.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:31.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:31.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:31.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:31.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:31.813 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:48:31.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:31.813785+0000) 2019-10-18T02:48:32.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:32.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:32.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:32.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:33.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:33.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:33.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:33.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:34.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:34.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:34.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:34.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:35.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:35.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:35.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:35.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:36.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:36.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:36.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:36.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:36.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:36.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:36.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:48:36.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:36.666205+0000) 2019-10-18T02:48:36.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:48:36.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:36.712648+0000) 2019-10-18T02:48:37.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:37.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:37.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:37.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:38.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:38.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:38.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:38.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:39.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:39.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:39.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:39.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:40.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:40.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:40.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:40.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:41.317 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:48:41.313+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:41.317480+0000) 2019-10-18T02:48:41.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:41.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:41.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:41.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:41.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:41.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:41.814 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:48:41.809+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:41.814015+0000) 2019-10-18T02:48:42.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:42.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:42.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:42.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:43.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:43.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:43.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:43.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:44.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:44.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:45.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:45.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:45.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:45.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:46.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:46.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:46.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:46.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:46.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:46.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:46.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:48:46.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:46.666297+0000) 2019-10-18T02:48:46.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:48:46.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:46.712750+0000) 2019-10-18T02:48:47.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:47.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:47.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:47.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:48.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:48.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:48.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:48.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:49.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:49.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:50.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:50.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:50.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:50.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:51.318 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:48:51.312+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:51.317671+0000) 2019-10-18T02:48:51.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:51.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:51.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:51.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:51.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:51.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:51.814 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:48:51.812+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:51.814280+0000) 2019-10-18T02:48:52.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:52.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:52.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:52.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:53.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:53.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:53.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:53.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:54.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:54.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:54.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:54.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:55.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:55.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:55.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:55.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:56.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:48:56.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:48:56.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:56.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:56.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:56.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:56.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:48:56.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:56.666401+0000) 2019-10-18T02:48:56.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:48:56.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:48:56.712849+0000) 2019-10-18T02:48:57.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:57.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:57.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:57.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:58.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:58.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:58.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:58.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:48:59.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:59.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:48:59.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:48:59.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:00.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:00.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:00.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:00.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:01.318 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:49:01.312+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:01.317847+0000) 2019-10-18T02:49:01.376 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:01.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:01.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:01.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:01.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:01.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:01.814 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:01.812+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:01.814466+0000) 2019-10-18T02:49:02.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:02.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:02.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:02.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:03.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:03.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:03.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:03.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:04.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:04.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:04.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:04.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:05.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:05.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:05.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:05.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:06.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:06.372+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:06.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:06.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:06.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:06.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:06.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:49:06.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:06.666504+0000) 2019-10-18T02:49:06.712 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:49:06.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:06.712945+0000) 2019-10-18T02:49:07.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:07.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:07.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:07.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:08.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:08.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:08.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:08.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:09.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:09.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:09.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:09.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:10.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:10.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:10.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:10.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:11.318 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:49:11.312+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:11.318030+0000) 2019-10-18T02:49:11.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:11.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:11.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:11.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:11.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:11.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:11.814 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:11.812+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:11.814678+0000) 2019-10-18T02:49:12.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:12.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:12.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:12.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:13.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:13.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:13.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:13.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:14.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:14.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:14.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:14.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:15.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:15.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:15.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:15.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:16.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:16.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:16.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:49:16.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:16.666610+0000) 2019-10-18T02:49:16.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:49:16.708+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:16.713047+0000) 2019-10-18T02:49:16.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:16.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:16.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:16.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:17.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:17.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:17.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:17.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:18.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:18.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:18.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:18.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:19.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:19.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:19.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:19.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:20.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:20.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:20.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:20.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:21.321 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:49:21.316+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:21.318215+0000) 2019-10-18T02:49:21.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:21.376+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:21.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:21.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:21.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:21.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:21.815 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:21.812+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:21.814890+0000) 2019-10-18T02:49:21.816 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:21.812+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:21.816158+0000) 2019-10-18T02:49:22.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:22.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:22.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:22.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:23.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:23.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:23.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:23.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:24.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:24.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:24.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:24.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:25.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:25.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:25.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:25.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:26.315 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:26.312+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:26.315400+0000) 2019-10-18T02:49:26.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:26.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:26.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:49:26.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:26.666722+0000) 2019-10-18T02:49:26.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:49:26.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:26.713147+0000) 2019-10-18T02:49:26.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:26.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:26.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:26.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:27.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:27.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:27.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:27.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:28.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:28.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:28.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:28.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:29.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:29.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:29.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:29.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:30.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:30.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:30.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:30.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:31.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:49:31.316+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:31.318371+0000) 2019-10-18T02:49:31.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:31.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:31.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:31.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:31.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:31.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:32.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:32.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:32.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:32.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:33.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:33.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:33.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:33.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:34.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:34.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:34.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:34.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:35.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:35.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:35.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:35.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:36.315 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:36.312+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:36.315579+0000) 2019-10-18T02:49:36.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:36.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:36.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:49:36.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:36.666823+0000) 2019-10-18T02:49:36.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:49:36.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:36.713244+0000) 2019-10-18T02:49:36.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:36.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:36.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:36.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:37.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:37.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:37.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:37.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:38.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:38.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:38.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:38.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:39.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:39.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:39.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:39.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:40.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:40.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:40.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:40.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:41.325 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:49:41.316+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:41.318548+0000) 2019-10-18T02:49:41.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:41.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:41.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:41.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:41.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:41.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:42.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:42.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:42.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:42.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:43.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:43.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:43.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:43.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:44.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:44.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:44.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:44.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:45.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:45.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:46.316 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:46.312+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:46.315763+0000) 2019-10-18T02:49:46.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:46.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:46.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:46.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:46.666 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:49:46.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:46.666930+0000) 2019-10-18T02:49:46.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:49:46.707+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:46.713344+0000) 2019-10-18T02:49:47.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:47.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:47.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:47.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:48.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:48.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:48.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:48.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:49.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:49.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:49.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:49.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:50.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:50.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:50.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:50.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:51.318 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:49:51.316+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:51.318738+0000) 2019-10-18T02:49:51.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:51.379+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:51.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:51.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:51.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:51.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:52.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:52.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:52.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:52.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:53.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:53.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:53.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:53.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:54.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:54.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:54.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:54.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:55.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:55.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:55.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:55.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:56.316 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:49:56.312+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:56.315947+0000) 2019-10-18T02:49:56.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:49:56.383+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:49:56.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:56.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:56.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:56.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:56.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:49:56.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:56.667035+0000) 2019-10-18T02:49:56.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:49:56.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:49:56.713449+0000) 2019-10-18T02:49:57.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:57.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:57.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:57.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:58.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:58.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:58.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:58.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:49:59.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:59.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:39:29.744540+0000 (oldest deadline 2019-10-18T02:39:49.744540+0000) 2019-10-18T02:49:59.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:49:59.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:00.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:00.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:01.319 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:50:01.316+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:01.318916+0000) 2019-10-18T02:50:01.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:01.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:01.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:01.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:02.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:02.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:03.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:03.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:04.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:04.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:05.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:05.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:06.316 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:50:06.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:06.316142+0000) 2019-10-18T02:50:06.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:06.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:06.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:06.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:06.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:50:06.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:06.667136+0000) 2019-10-18T02:50:06.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:50:06.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:06.713545+0000) 2019-10-18T02:50:07.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:07.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:08.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:08.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:09.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:09.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:10.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:10.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:11.319 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:50:11.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:11.319108+0000) 2019-10-18T02:50:11.388 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:11.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:11.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:11.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:12.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:12.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:13.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:13.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:14.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:14.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:15.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:15.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:16.317 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:50:16.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:16.316329+0000) 2019-10-18T02:50:16.389 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:16.382+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:16.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:16.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:16.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:50:16.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:16.667237+0000) 2019-10-18T02:50:16.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:50:16.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:16.713639+0000) 2019-10-18T02:50:17.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:17.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:18.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:18.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:19.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:19.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:20.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:20.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:21.319 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:50:21.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:21.319280+0000) 2019-10-18T02:50:21.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:21.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:21.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:21.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:21.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:21.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:22.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:22.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:22.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:22.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:23.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:23.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:23.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:23.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:24.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:24.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:24.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:24.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:25.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:25.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:25.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:25.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:26.316 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:50:26.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:26.316521+0000) 2019-10-18T02:50:26.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:26.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:26.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:26.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:26.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:26.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:26.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:50:26.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:26.667345+0000) 2019-10-18T02:50:26.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:50:26.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:26.713738+0000) 2019-10-18T02:50:27.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:27.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:27.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:28.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:28.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:28.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:28.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:29.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:29.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:29.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:29.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:30.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:30.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:30.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:30.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:31.320 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:50:31.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:31.319484+0000) 2019-10-18T02:50:31.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:31.386+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:31.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:31.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:32.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:32.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:33.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:33.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:33.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:33.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:34.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:34.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:34.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:34.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:35.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:35.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:35.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:35.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:36.316 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:50:36.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:36.316676+0000) 2019-10-18T02:50:36.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:36.385+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:36.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:36.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:36.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:36.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:36.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:50:36.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:36.667456+0000) 2019-10-18T02:50:36.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:50:36.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:36.713838+0000) 2019-10-18T02:50:37.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:37.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:37.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:37.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:38.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:38.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:38.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:38.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:39.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:39.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:39.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:39.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:40.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:40.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:40.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:40.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:41.319 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:50:41.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:41.319683+0000) 2019-10-18T02:50:41.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:41.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:41.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:41.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:41.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:41.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:42.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:42.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:42.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:42.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:43.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:43.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:43.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:43.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:44.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:44.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:44.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:44.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:45.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:45.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:45.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:45.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:46.317 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:50:46.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:46.316866+0000) 2019-10-18T02:50:46.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:46.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:46.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:50:46.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:46.667556+0000) 2019-10-18T02:50:46.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:50:46.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:46.713935+0000) 2019-10-18T02:50:46.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:46.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:46.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:46.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:47.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:47.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:47.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:47.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:48.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:48.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:48.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:48.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:49.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:49.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:49.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:49.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:50.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:50.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:50.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:50.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:51.319 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:50:51.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:51.319833+0000) 2019-10-18T02:50:51.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:51.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:51.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:51.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:51.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:51.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:52.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:52.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:52.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:52.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:53.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:53.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:53.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:53.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:54.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:54.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:54.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:54.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:55.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:55.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:55.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:55.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:56.317 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:50:56.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:56.317046+0000) 2019-10-18T02:50:56.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:50:56.393+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:50:56.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:50:56.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:56.667659+0000) 2019-10-18T02:50:56.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:56.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:56.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:56.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:56.713 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:50:56.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:50:56.714030+0000) 2019-10-18T02:50:57.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:57.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:57.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:57.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:58.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:58.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:58.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:58.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:50:59.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:59.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:50:59.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:50:59.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:00.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:00.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:00.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:00.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:01.320 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:51:01.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:01.320012+0000) 2019-10-18T02:51:01.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:01.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:01.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:01.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:01.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:01.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:02.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:02.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:02.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:02.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:03.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:03.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:03.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:03.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:04.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:04.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:04.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:04.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:05.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:05.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:05.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:05.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:06.317 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:51:06.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:06.317229+0000) 2019-10-18T02:51:06.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:06.397+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:06.667 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:51:06.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:06.667758+0000) 2019-10-18T02:51:06.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:51:06.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:06.714126+0000) 2019-10-18T02:51:06.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:06.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:06.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:06.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:07.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:07.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:07.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:07.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:08.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:08.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:08.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:08.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:09.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:09.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:10.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:10.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:10.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:10.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:11.320 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:51:11.315+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:11.320225+0000) 2019-10-18T02:51:11.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:11.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:11.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:11.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:11.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:11.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:12.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:12.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:13.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:13.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:13.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:13.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:14.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:14.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:14.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:14.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:15.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:15.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:15.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:16.317 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:51:16.311+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:16.317416+0000) 2019-10-18T02:51:16.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:16.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:16.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:51:16.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:16.667861+0000) 2019-10-18T02:51:16.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:51:16.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:16.714227+0000) 2019-10-18T02:51:16.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:16.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:16.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:16.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:17.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:17.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:17.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:17.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:18.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:18.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:19.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:19.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:19.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:19.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:20.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:20.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:20.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:20.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:21.320 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:51:21.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:21.320408+0000) 2019-10-18T02:51:21.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:21.396+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:21.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:21.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:21.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:21.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:23.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:23.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:23.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:23.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:23.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:23.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:23.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:25.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:25.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:25.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:25.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:26.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:26.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:26.317 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:51:26.310+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:26.317598+0000) 2019-10-18T02:51:26.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:26.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:26.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:51:26.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:26.667964+0000) 2019-10-18T02:51:26.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:51:26.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:26.714327+0000) 2019-10-18T02:51:27.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:27.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:27.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:27.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:28.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:28.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:28.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:28.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:29.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:29.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:29.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:29.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:30.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:30.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:30.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:30.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:31.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:31.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:31.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:31.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:31.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:51:31.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:31.320591+0000) 2019-10-18T02:51:31.403 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:31.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:32.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:32.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:32.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:32.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:33.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:33.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:33.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:33.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:34.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:34.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:34.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:34.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:35.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:35.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:35.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:35.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:36.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:36.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:36.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:36.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:36.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:51:36.310+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:36.317801+0000) 2019-10-18T02:51:36.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:36.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:36.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:51:36.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:36.668074+0000) 2019-10-18T02:51:36.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:51:36.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:36.714429+0000) 2019-10-18T02:51:37.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:37.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:37.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:37.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:38.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:38.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:38.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:38.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:39.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:39.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:39.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:39.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:40.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:40.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:40.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:40.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:41.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:41.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:41.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:41.321 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:51:41.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:41.320804+0000) 2019-10-18T02:51:41.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:41.400+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:42.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:42.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:42.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:42.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:43.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:43.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:43.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:43.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:44.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:44.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:44.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:44.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:45.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:45.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:45.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:45.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:46.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:45.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:46.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:45.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:46.318 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:51:46.310+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:46.318020+0000) 2019-10-18T02:51:46.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:46.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:46.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:51:46.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:46.668189+0000) 2019-10-18T02:51:46.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:51:46.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:46.714534+0000) 2019-10-18T02:51:46.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:46.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:46.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:46.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:48.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:48.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:48.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:48.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:49.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:49.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:49.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:49.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:50.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:50.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:50.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:50.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:51.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:51.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:51.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:51.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:51.321 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:51:51.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:51.320973+0000) 2019-10-18T02:51:51.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:51.403+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:52.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:52.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:52.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:52.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:53.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:53.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:53.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:53.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:54.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:54.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:54.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:54.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:55.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:55.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:55.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:55.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:56.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:56.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:56.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:56.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:56.318 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:51:56.314+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:56.318206+0000) 2019-10-18T02:51:56.410 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:51:56.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:51:56.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:51:56.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:56.668305+0000) 2019-10-18T02:51:56.714 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:51:56.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:51:56.714629+0000) 2019-10-18T02:51:57.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:57.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:57.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:57.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:58.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:58.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:58.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:58.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:51:59.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:59.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:51:59.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:51:59.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:00.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:00.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:00.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:00.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:01.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:01.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:01.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:01.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:01.321 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:52:01.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:01.321181+0000) 2019-10-18T02:52:01.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:01.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:02.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:02.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:02.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:02.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:03.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:03.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:03.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:03.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:04.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:04.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:04.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:04.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:05.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:05.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:05.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:05.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:06.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:06.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:06.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:06.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:06.318 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:52:06.314+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:06.318353+0000) 2019-10-18T02:52:06.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:06.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:06.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:52:06.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:06.668408+0000) 2019-10-18T02:52:06.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:52:06.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:06.714728+0000) 2019-10-18T02:52:07.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:07.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:07.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:07.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:08.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:08.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:08.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:08.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:09.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:09.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:09.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:09.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:10.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:10.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:10.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:10.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:11.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:11.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:11.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:11.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:11.321 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:52:11.314+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:11.321390+0000) 2019-10-18T02:52:11.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:11.407+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:12.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:12.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:12.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:12.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:13.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:13.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:13.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:13.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:14.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:14.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:15.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:15.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:15.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:15.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:16.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:16.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:16.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:16.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:16.318 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:52:16.314+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:16.318504+0000) 2019-10-18T02:52:16.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:16.411+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:16.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:52:16.663+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:16.668512+0000) 2019-10-18T02:52:16.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:52:16.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:16.714829+0000) 2019-10-18T02:52:17.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:17.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:17.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:17.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:18.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:18.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:18.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:18.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:19.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:19.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:20.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:20.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:20.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:20.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:21.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:21.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:21.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:21.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:21.321 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:52:21.318+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:21.321600+0000) 2019-10-18T02:52:21.415 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:21.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:22.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:22.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:22.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:22.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:23.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:23.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:23.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:23.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:24.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:24.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:24.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:24.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:25.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:25.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:25.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:25.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:26.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:26.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:26.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:26.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:26.318 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:52:26.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:26.318670+0000) 2019-10-18T02:52:26.415 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:26.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:26.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:52:26.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:26.668620+0000) 2019-10-18T02:52:26.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:52:26.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:26.714928+0000) 2019-10-18T02:52:27.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:27.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:27.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:27.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:28.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:28.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:28.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:28.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:29.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:29.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:29.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:29.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:30.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:30.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:30.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:30.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:31.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:31.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:31.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:31.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:31.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:52:31.317+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:31.321812+0000) 2019-10-18T02:52:31.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:31.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:32.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:32.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:32.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:32.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:33.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:33.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:33.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:33.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:34.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:34.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:34.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:34.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:35.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:35.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:35.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:35.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:36.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:36.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:36.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:36.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:36.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:52:36.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:36.318855+0000) 2019-10-18T02:52:36.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:36.410+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:36.668 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:52:36.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:36.668721+0000) 2019-10-18T02:52:36.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:52:36.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:36.715044+0000) 2019-10-18T02:52:37.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:37.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:37.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:37.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:38.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:38.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:38.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:38.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:39.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:39.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:39.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:39.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:40.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:40.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:40.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:40.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:41.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:41.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:41.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:41.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:41.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:52:41.317+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:41.321996+0000) 2019-10-18T02:52:41.418 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:41.414+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:42.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:42.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:42.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:42.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:43.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:43.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:43.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:43.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:44.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:44.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:44.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:44.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:45.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:45.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:45.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:45.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:46.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:46.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:46.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:46.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:46.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:52:46.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:46.319065+0000) 2019-10-18T02:52:46.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:46.414+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:46.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:52:46.662+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:46.668832+0000) 2019-10-18T02:52:46.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:52:46.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:46.715147+0000) 2019-10-18T02:52:46.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:46.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:46.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:46.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:48.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:48.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:48.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:48.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:48.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:48.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:48.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:48.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:49.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:49.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:49.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:49.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:50.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:50.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:50.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:50.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:51.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:52:51.321+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:51.322215+0000) 2019-10-18T02:52:51.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:51.414+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:51.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:51.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:51.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:51.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:52.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:52.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:52.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:52.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:53.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:53.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:53.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:53.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:54.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:54.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:54.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:54.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:55.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:55.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:55.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:55.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:56.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:52:56.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:56.319249+0000) 2019-10-18T02:52:56.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:52:56.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:52:56.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:52:56.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:56.668949+0000) 2019-10-18T02:52:56.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:52:56.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:52:56.715252+0000) 2019-10-18T02:52:57.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:56.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:57.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:56.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:57.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:57.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:57.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:57.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:58.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:58.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:58.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:58.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:52:59.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:59.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:52:59.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:52:59.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:01.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:00.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:01.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:00.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:01.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:53:01.321+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:01.322382+0000) 2019-10-18T02:53:01.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:01.417+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:01.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:01.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:01.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:01.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:02.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:02.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:02.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:02.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:03.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:03.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:04.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:04.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:04.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:04.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:05.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:05.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:05.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:05.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:06.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:53:06.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:06.319430+0000) 2019-10-18T02:53:06.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:06.417+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:06.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:53:06.661+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:06.669053+0000) 2019-10-18T02:53:06.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:53:06.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:06.715352+0000) 2019-10-18T02:53:07.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:07.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:07.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:07.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:07.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:07.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:07.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:07.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:09.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:09.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:09.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:09.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:10.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:10.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:10.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:10.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:10.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:10.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:10.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:10.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:11.322 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:53:11.321+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:11.322564+0000) 2019-10-18T02:53:11.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:11.417+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:11.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:11.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:11.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:11.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:12.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:12.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:12.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:12.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:13.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:13.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:13.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:13.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:14.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:14.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:14.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:14.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:15.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:15.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:15.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:15.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:16.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:53:16.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:16.319615+0000) 2019-10-18T02:53:16.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:16.421+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:16.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:53:16.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:16.669158+0000) 2019-10-18T02:53:16.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:53:16.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:16.715450+0000) 2019-10-18T02:53:16.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:16.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:16.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:16.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:17.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:17.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:17.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:17.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:18.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:18.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:18.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:18.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:19.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:19.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:19.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:19.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:20.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:20.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:20.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:20.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:21.323 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:53:21.321+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:21.322773+0000) 2019-10-18T02:53:21.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:21.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:21.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:21.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:21.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:21.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:22.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:22.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:22.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:22.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:23.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:23.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:23.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:23.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:24.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:24.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:24.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:24.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:25.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:25.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:25.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:25.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:26.319 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:53:26.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:26.319795+0000) 2019-10-18T02:53:26.427 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:26.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:26.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:53:26.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:26.669264+0000) 2019-10-18T02:53:26.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:53:26.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:26.715549+0000) 2019-10-18T02:53:26.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:26.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:26.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:26.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:27.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:27.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:27.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:27.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:28.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:28.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:28.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:28.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:29.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:29.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:29.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:29.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:30.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:30.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:30.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:30.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:31.325 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:53:31.321+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:31.322980+0000) 2019-10-18T02:53:31.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:31.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:31.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:31.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:31.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:31.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:32.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:32.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:32.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:32.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:33.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:33.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:33.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:33.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:34.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:34.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:35.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:35.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:35.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:35.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:36.320 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:53:36.317+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:36.319975+0000) 2019-10-18T02:53:36.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:36.424+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:36.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:53:36.664+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:36.669365+0000) 2019-10-18T02:53:36.715 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:53:36.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:36.715647+0000) 2019-10-18T02:53:36.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:36.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:36.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:36.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:37.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:37.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:37.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:37.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:38.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:38.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:38.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:38.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:39.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:39.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:39.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:39.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:40.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:40.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:40.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:40.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:41.323 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:53:41.321+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:41.323196+0000) 2019-10-18T02:53:41.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:41.428+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:41.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:41.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:41.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:41.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:42.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:42.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:42.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:42.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:43.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:43.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:43.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:43.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:44.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:44.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:44.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:44.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:45.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:45.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:45.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:45.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:46.320 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:53:46.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:46.320151+0000) 2019-10-18T02:53:46.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:46.428+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:46.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:53:46.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:46.669468+0000) 2019-10-18T02:53:46.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:53:46.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:46.715749+0000) 2019-10-18T02:53:46.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:46.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:46.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:46.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:47.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:47.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:47.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:47.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:48.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:48.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:48.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:48.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:49.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:49.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:49.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:49.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:50.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:50.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:50.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:50.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:51.324 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:53:51.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:51.323411+0000) 2019-10-18T02:53:51.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:51.428+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:51.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:51.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:51.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:51.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:52.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:52.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:52.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:52.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:53.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:53.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:53.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:53.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:54.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:54.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:54.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:54.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:55.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:55.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:55.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:55.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:56.320 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:53:56.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:56.320362+0000) 2019-10-18T02:53:56.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:53:56.428+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:53:56.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:56.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:56.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:56.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:56.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:53:56.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:56.669576+0000) 2019-10-18T02:53:56.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:53:56.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:53:56.715850+0000) 2019-10-18T02:53:57.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:57.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:57.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:57.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:58.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:58.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:58.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:58.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:53:59.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:59.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:53:59.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:53:59.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:00.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:00.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:00.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:00.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:01.323 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:54:01.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:01.323626+0000) 2019-10-18T02:54:01.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:01.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:01.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:01.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:01.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:01.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:02.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:02.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:02.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:02.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:03.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:03.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:04.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:04.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:04.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:04.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:05.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:05.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:05.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:05.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:06.320 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:54:06.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:06.320575+0000) 2019-10-18T02:54:06.434 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:06.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:06.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:54:06.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:06.669679+0000) 2019-10-18T02:54:06.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:54:06.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:06.715953+0000) 2019-10-18T02:54:06.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:06.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:06.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:06.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:07.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:07.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:07.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:07.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:08.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:08.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:08.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:08.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:09.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:09.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:09.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:09.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:10.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:10.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:10.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:10.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:11.324 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:54:11.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:11.323810+0000) 2019-10-18T02:54:11.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:11.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:11.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:11.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:11.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:11.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:12.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:12.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:12.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:12.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:13.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:13.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:13.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:13.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:14.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:14.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:14.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:14.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:15.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:15.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:15.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:15.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:16.321 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:54:16.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:16.320792+0000) 2019-10-18T02:54:16.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:16.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:16.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:54:16.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:16.669783+0000) 2019-10-18T02:54:16.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:54:16.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:16.716056+0000) 2019-10-18T02:54:16.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:16.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:16.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:16.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:17.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:17.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:17.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:17.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:18.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:18.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:18.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:18.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:19.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:19.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:19.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:19.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:20.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:20.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:20.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:20.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:21.324 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:54:21.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:21.323997+0000) 2019-10-18T02:54:21.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:21.431+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:21.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:21.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:21.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:21.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:22.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:22.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:22.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:22.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:23.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:23.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:23.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:23.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:24.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:24.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:24.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:24.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:25.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:25.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:25.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:25.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:26.321 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:54:26.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:26.320997+0000) 2019-10-18T02:54:26.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:26.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:26.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:54:26.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:26.669888+0000) 2019-10-18T02:54:26.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:54:26.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:26.716156+0000) 2019-10-18T02:54:26.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:26.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:26.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:26.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:27.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:27.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:28.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:28.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:28.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:28.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:29.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:29.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:29.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:29.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:30.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:30.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:30.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:30.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:31.324 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:54:31.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:31.324205+0000) 2019-10-18T02:54:31.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:31.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:31.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:31.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:31.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:31.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:32.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:32.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:32.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:32.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:33.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:33.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:33.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:33.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:34.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:34.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:34.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:34.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:35.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:35.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:35.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:35.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:36.321 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:54:36.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:36.321210+0000) 2019-10-18T02:54:36.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:36.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:36.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:36.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:36.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:36.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:36.669 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:54:36.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:36.669987+0000) 2019-10-18T02:54:36.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:54:36.711+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:36.716257+0000) 2019-10-18T02:54:37.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:37.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:37.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:37.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:38.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:38.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:38.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:38.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:39.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:39.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:39.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:39.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:40.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:40.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:40.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:40.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:41.324 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:54:41.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:41.324395+0000) 2019-10-18T02:54:41.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:41.435+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:41.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:41.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:41.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:41.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:42.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:42.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:42.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:42.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:43.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:43.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:43.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:43.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:44.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:44.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:44.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:44.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:45.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:45.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:45.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:45.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:46.321 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:54:46.316+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:46.321393+0000) 2019-10-18T02:54:46.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:46.434+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:46.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:54:46.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:46.670097+0000) 2019-10-18T02:54:46.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:54:46.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:46.716361+0000) 2019-10-18T02:54:46.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:46.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:46.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:46.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:47.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:47.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:47.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:47.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:48.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:48.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:48.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:48.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:49.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:49.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:49.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:49.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:50.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:50.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:50.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:50.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:51.325 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:54:51.320+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:51.324561+0000) 2019-10-18T02:54:51.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:51.438+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:51.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:51.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:51.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:51.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:52.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:52.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:52.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:52.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:53.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:53.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:53.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:53.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:54.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:54.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:54.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:54.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:55.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:55.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:55.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:55.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:56.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:54:56.315+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:56.321562+0000) 2019-10-18T02:54:56.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:54:56.438+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:54:56.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:54:56.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:56.670200+0000) 2019-10-18T02:54:56.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:54:56.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:54:56.716454+0000) 2019-10-18T02:54:56.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:56.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:56.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:56.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:57.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:57.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:57.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:57.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:58.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:58.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:58.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:58.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:54:59.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:59.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:54:59.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:54:59.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:00.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:00.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:00.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:00.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:01.324 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:01.319+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:01.324736+0000) 2019-10-18T02:55:01.326 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:01.319+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:01.325783+0000) 2019-10-18T02:55:01.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:01.438+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:01.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:01.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:01.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:01.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:02.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:02.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:02.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:02.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:04.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:04.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:04.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:04.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:05.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:05.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:05.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:05.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:05.825 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:05.819+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:05.825174+0000) 2019-10-18T02:55:06.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:06.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:06.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:06.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:06.321 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:55:06.315+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:06.321730+0000) 2019-10-18T02:55:06.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:06.438+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:06.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:55:06.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:06.670305+0000) 2019-10-18T02:55:06.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:55:06.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:06.716555+0000) 2019-10-18T02:55:07.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:06.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:07.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:06.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:08.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:08.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:08.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:08.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:09.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:09.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:09.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:09.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:10.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:10.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:10.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:10.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:11.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:11.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:11.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:11.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:11.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:11.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:12.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:12.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:12.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:12.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:13.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:13.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:13.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:13.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:14.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:14.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:14.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:14.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:15.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:15.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:15.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:15.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:15.825 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:15.819+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:15.825356+0000) 2019-10-18T02:55:16.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:16.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:16.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:16.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:16.322 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:55:16.315+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:16.321915+0000) 2019-10-18T02:55:16.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:16.442+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:16.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:55:16.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:16.670420+0000) 2019-10-18T02:55:16.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:55:16.710+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:16.716660+0000) 2019-10-18T02:55:17.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:16.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:17.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:16.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:18.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:18.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:18.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:18.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:18.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:18.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:18.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:18.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:19.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:19.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:19.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:19.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:20.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:20.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:20.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:20.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:21.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:21.441+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:21.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:21.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:21.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:21.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:23.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:22.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:23.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:22.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:24.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:24.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:24.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:24.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:24.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:24.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:24.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:24.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:25.825 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:25.819+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:25.825534+0000) 2019-10-18T02:55:26.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:26.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:26.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:26.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:26.322 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:55:26.315+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:26.322096+0000) 2019-10-18T02:55:26.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:26.441+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:26.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:55:26.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:26.670526+0000) 2019-10-18T02:55:26.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:55:26.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:26.716759+0000) 2019-10-18T02:55:27.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:27.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:27.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:27.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:28.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:28.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:28.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:28.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:29.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:29.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:29.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:29.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:30.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:29.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:30.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:29.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:30.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:30.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:30.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:30.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:31.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:31.441+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:31.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:31.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:31.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:31.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:32.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:32.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:33.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:33.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:33.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:33.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:34.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:34.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:34.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:34.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:35.825 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:35.819+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:35.825719+0000) 2019-10-18T02:55:36.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:35.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:36.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:36.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:36.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:55:36.319+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:36.322259+0000) 2019-10-18T02:55:36.450 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:36.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:36.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:55:36.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:36.670626+0000) 2019-10-18T02:55:36.716 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:55:36.709+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:36.716859+0000) 2019-10-18T02:55:37.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:37.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:37.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:37.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:37.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:37.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:37.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:37.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:38.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:38.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:38.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:38.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:39.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:39.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:39.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:39.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:40.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:40.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:40.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:40.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:41.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:41.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:41.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:41.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:41.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:41.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:42.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:42.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:42.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:42.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:43.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:43.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:43.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:43.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:44.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:44.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:44.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:44.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:45.826 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:45.819+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:45.825899+0000) 2019-10-18T02:55:45.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:45.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:45.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:45.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:46.322 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:55:46.319+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:46.322438+0000) 2019-10-18T02:55:46.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:46.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:46.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:55:46.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:46.670732+0000) 2019-10-18T02:55:46.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:55:46.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:46.716962+0000) 2019-10-18T02:55:46.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:46.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:46.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:46.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:47.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:47.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:47.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:47.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:48.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:48.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:48.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:48.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:49.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:49.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:49.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:49.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:50.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:50.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:50.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:50.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:51.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:51.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:51.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:51.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:51.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:51.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:52.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:52.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:52.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:52.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:53.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:53.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:53.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:53.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:54.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:54.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:54.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:54.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:55.826 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:55:55.819+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:55.826106+0000) 2019-10-18T02:55:55.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:55.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:55.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:55.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:56.325 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:55:56.319+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:56.322592+0000) 2019-10-18T02:55:56.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:55:56.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:55:56.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:55:56.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:56.670837+0000) 2019-10-18T02:55:56.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:55:56.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:55:56.717059+0000) 2019-10-18T02:55:56.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:56.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:56.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:56.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:57.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:57.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:57.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:57.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:58.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:58.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:58.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:58.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:55:59.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:59.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:55:59.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:55:59.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:00.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:00.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:00.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:00.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:01.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:01.452+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:01.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:01.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:01.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:01.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:02.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:02.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:02.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:02.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:03.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:03.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:03.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:03.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:04.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:04.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:04.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:04.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:05.826 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:56:05.823+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:05.826260+0000) 2019-10-18T02:56:05.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:05.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:05.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:05.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:06.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:56:06.319+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:06.322766+0000) 2019-10-18T02:56:06.455 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:06.452+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:06.670 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:56:06.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:06.670938+0000) 2019-10-18T02:56:06.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:56:06.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:06.717156+0000) 2019-10-18T02:56:06.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:06.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:06.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:06.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:07.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:07.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:07.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:07.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:08.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:08.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:08.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:08.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:09.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:09.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:09.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:09.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:10.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:10.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:10.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:10.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:11.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:11.452+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:11.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:11.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:12.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:12.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:12.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:12.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:13.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:13.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:13.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:13.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:14.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:14.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:14.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:14.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:15.826 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:56:15.822+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:15.826424+0000) 2019-10-18T02:56:15.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:15.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:15.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:15.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:16.323 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:56:16.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:16.322943+0000) 2019-10-18T02:56:16.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:16.452+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:16.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:56:16.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:16.671051+0000) 2019-10-18T02:56:16.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:56:16.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:16.717260+0000) 2019-10-18T02:56:16.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:16.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:16.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:16.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:17.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:17.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:17.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:17.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:18.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:18.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:18.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:18.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:19.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:19.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:19.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:19.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:20.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:20.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:20.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:20.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:21.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:21.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:21.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:21.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:21.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:21.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:22.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:22.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:22.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:22.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:23.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:23.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:23.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:23.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:24.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:24.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:24.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:24.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:25.826 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:56:25.822+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:25.826565+0000) 2019-10-18T02:56:25.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:25.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:25.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:25.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:26.323 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:56:26.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:26.323132+0000) 2019-10-18T02:56:26.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:26.456+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:26.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:56:26.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:26.671167+0000) 2019-10-18T02:56:26.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:56:26.712+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:26.717368+0000) 2019-10-18T02:56:26.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:26.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:26.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:26.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:27.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:27.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:27.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:27.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:28.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:28.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:28.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:28.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:29.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:29.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:29.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:29.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:30.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:30.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:30.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:30.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:31.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:31.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:31.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:31.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:31.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:31.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:32.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:32.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:32.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:32.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:33.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:33.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:33.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:33.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:34.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:34.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:34.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:34.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:35.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:35.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:35.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:35.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:35.826 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:56:35.822+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:35.826743+0000) 2019-10-18T02:56:36.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:56:36.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:36.323312+0000) 2019-10-18T02:56:36.460 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:36.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:36.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:56:36.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:36.671284+0000) 2019-10-18T02:56:36.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:36.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:36.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:36.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:36.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:56:36.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:36.717462+0000) 2019-10-18T02:56:37.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:37.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:37.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:37.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:38.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:38.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:38.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:38.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:39.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:39.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:39.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:39.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:40.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:40.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:40.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:40.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:41.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:41.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:41.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:41.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:41.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:41.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:42.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:42.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:42.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:42.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:43.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:43.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:43.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:43.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:44.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:44.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:44.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:44.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:45.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:45.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:45.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:45.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:45.827 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:56:45.822+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:45.826923+0000) 2019-10-18T02:56:46.323 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:56:46.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:46.323509+0000) 2019-10-18T02:56:46.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:46.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:46.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:56:46.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:46.671390+0000) 2019-10-18T02:56:46.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:56:46.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:46.717559+0000) 2019-10-18T02:56:46.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:46.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:46.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:46.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:47.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:47.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:47.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:47.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:48.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:48.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:48.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:48.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:49.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:49.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:49.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:49.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:50.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:50.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:50.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:50.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:51.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:51.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:51.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:51.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:51.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:51.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:52.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:52.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:52.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:52.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:53.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:53.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:53.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:53.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:54.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:54.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:54.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:54.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:55.827 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:56:55.822+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:55.827105+0000) 2019-10-18T02:56:55.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:55.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:55.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:55.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:56.323 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:56:56.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:56.323703+0000) 2019-10-18T02:56:56.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:56:56.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:56:56.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:56:56.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:56.671493+0000) 2019-10-18T02:56:56.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:56:56.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:56:56.717655+0000) 2019-10-18T02:56:56.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:56.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:56.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:56.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:57.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:57.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:57.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:57.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:58.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:58.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:58.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:58.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:56:59.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:59.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:56:59.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:56:59.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:00.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:00.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:00.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:00.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:01.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:01.459+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:01.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:01.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:01.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:01.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:02.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:02.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:02.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:02.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:03.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:03.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:03.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:03.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:04.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:04.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:04.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:04.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:05.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:05.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:05.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:05.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:05.827 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:57:05.822+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:05.827289+0000) 2019-10-18T02:57:06.325 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:57:06.318+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:06.323889+0000) 2019-10-18T02:57:06.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:06.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:06.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:57:06.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:06.671596+0000) 2019-10-18T02:57:06.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:57:06.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:06.717762+0000) 2019-10-18T02:57:06.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:06.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:06.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:06.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:07.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:07.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:07.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:07.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:08.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:08.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:08.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:08.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:09.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:09.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:09.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:09.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:10.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:10.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:10.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:10.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:11.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:11.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:11.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:11.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:11.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:11.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:12.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:12.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:12.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:12.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:13.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:13.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:13.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:13.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:14.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:14.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:14.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:14.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:15.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:15.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:15.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:15.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:15.827 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:57:15.826+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:15.827469+0000) 2019-10-18T02:57:16.326 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:57:16.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:16.324073+0000) 2019-10-18T02:57:16.467 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:16.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:16.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:16.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:16.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:16.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:16.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:57:16.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:16.671700+0000) 2019-10-18T02:57:16.717 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:57:16.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:16.717872+0000) 2019-10-18T02:57:17.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:17.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:17.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:17.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:18.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:18.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:18.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:18.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:19.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:19.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:19.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:19.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:20.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:20.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:20.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:20.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:21.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:21.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:21.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:21.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:21.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:21.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:22.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:22.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:22.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:22.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:23.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:23.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:23.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:23.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:24.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:24.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:24.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:24.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:25.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:25.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:25.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:25.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:25.827 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:57:25.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:25.827650+0000) 2019-10-18T02:57:26.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:57:26.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:26.324255+0000) 2019-10-18T02:57:26.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:26.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:26.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:26.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:26.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:26.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:26.671 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:57:26.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:26.671813+0000) 2019-10-18T02:57:26.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:57:26.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:26.717975+0000) 2019-10-18T02:57:27.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:27.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:27.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:27.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:28.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:28.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:28.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:28.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:29.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:29.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:29.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:29.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:30.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:30.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:30.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:30.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:31.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:31.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:31.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:31.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:31.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:31.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:32.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:32.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:33.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:33.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:33.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:33.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:34.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:34.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:34.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:34.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:35.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:35.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:35.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:35.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:35.830 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:57:35.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:35.827856+0000) 2019-10-18T02:57:36.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:57:36.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:36.324467+0000) 2019-10-18T02:57:36.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:36.466+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:36.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:36.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:36.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:36.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:36.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:57:36.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:36.671926+0000) 2019-10-18T02:57:36.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:57:36.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:36.718079+0000) 2019-10-18T02:57:37.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:37.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:37.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:37.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:38.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:38.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:38.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:38.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:39.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:39.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:39.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:39.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:40.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:40.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:40.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:40.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:41.472 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:41.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:41.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:41.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:41.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:41.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:42.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:42.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:42.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:42.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:43.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:43.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:43.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:43.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:44.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:44.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:44.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:44.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:45.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:45.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:45.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:45.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:45.828 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:57:45.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:45.827989+0000) 2019-10-18T02:57:46.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:57:46.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:46.324645+0000) 2019-10-18T02:57:46.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:46.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:46.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:46.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:46.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:46.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:46.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:57:46.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:46.672024+0000) 2019-10-18T02:57:46.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:57:46.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:46.718176+0000) 2019-10-18T02:57:47.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:47.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:47.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:47.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:48.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:48.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:48.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:48.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:49.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:49.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:49.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:49.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:50.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:50.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:50.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:50.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:51.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:51.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:51.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:51.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:51.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:51.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:52.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:52.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:52.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:52.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:53.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:53.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:53.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:53.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:54.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:54.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:54.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:54.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:55.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:55.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:55.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:55.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:55.828 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:57:55.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:55.828175+0000) 2019-10-18T02:57:56.324 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:57:56.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:56.324860+0000) 2019-10-18T02:57:56.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:57:56.469+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:57:56.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:56.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:56.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:56.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:56.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:57:56.665+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:56.672129+0000) 2019-10-18T02:57:56.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:57:56.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:57:56.718272+0000) 2019-10-18T02:57:57.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:57.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:57.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:57.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:58.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:58.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:57:59.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:59.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:57:59.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:57:59.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:00.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:00.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:00.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:00.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:01.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:01.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:01.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:01.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:01.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:01.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:02.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:02.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:02.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:02.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:03.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:03.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:03.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:03.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:04.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:04.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:04.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:04.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:05.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:05.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:05.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:05.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:05.828 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:58:05.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:05.828392+0000) 2019-10-18T02:58:06.325 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:58:06.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:06.325042+0000) 2019-10-18T02:58:06.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:06.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:06.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:58:06.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:06.672235+0000) 2019-10-18T02:58:06.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:58:06.717+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:06.718370+0000) 2019-10-18T02:58:06.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:06.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:06.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:06.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:07.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:07.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:07.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:07.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:08.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:08.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:08.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:08.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:09.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:09.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:09.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:09.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:10.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:10.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:10.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:10.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:11.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:11.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:11.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:11.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:11.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:11.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:12.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:12.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:12.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:12.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:13.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:13.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:13.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:13.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:14.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:14.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:14.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:14.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:15.828 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:58:15.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:15.828576+0000) 2019-10-18T02:58:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:15.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:15.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:15.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:16.325 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:58:16.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:16.325225+0000) 2019-10-18T02:58:16.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:16.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:16.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:58:16.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:16.672337+0000) 2019-10-18T02:58:16.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:58:16.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:16.718472+0000) 2019-10-18T02:58:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:16.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:16.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:16.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:17.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:17.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:17.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:17.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:18.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:18.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:18.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:19.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:19.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:19.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:19.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:20.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:20.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:20.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:20.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:21.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:21.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:21.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:21.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:21.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:21.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:22.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:22.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:22.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:22.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:23.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:23.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:23.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:23.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:24.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:24.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:25.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:25.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:25.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:25.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:25.828 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:58:25.825+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:25.828755+0000) 2019-10-18T02:58:26.325 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:58:26.321+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:26.325405+0000) 2019-10-18T02:58:26.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:26.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:26.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:26.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:26.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:26.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:26.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:58:26.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:26.672437+0000) 2019-10-18T02:58:26.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:58:26.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:26.718572+0000) 2019-10-18T02:58:27.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:27.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:27.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:27.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:28.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:28.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:28.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:28.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:29.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:29.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:29.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:29.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:30.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:30.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:30.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:30.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:31.479 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:31.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:31.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:31.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:31.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:31.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:32.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:32.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:32.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:32.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:33.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:33.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:33.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:34.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:34.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:34.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:34.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:35.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:35.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:35.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:35.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:35.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:58:35.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:35.828936+0000) 2019-10-18T02:58:36.325 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:58:36.320+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:36.325620+0000) 2019-10-18T02:58:36.481 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:36.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:36.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:58:36.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:36.672546+0000) 2019-10-18T02:58:36.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:58:36.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:36.718676+0000) 2019-10-18T02:58:36.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:36.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:36.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:36.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:37.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:37.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:37.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:37.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:38.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:38.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:38.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:38.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:39.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:39.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:39.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:39.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:40.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:40.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:40.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:40.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:41.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:41.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:41.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:41.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:41.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:41.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:42.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:42.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:42.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:42.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:43.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:43.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:43.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:43.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:44.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:44.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:44.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:44.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:45.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:45.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:45.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:45.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:45.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:58:45.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:45.829117+0000) 2019-10-18T02:58:46.328 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:58:46.320+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:46.325801+0000) 2019-10-18T02:58:46.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:46.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:46.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:58:46.668+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:46.672657+0000) 2019-10-18T02:58:46.718 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:58:46.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:46.718782+0000) 2019-10-18T02:58:46.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:46.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:46.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:46.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:47.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:47.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:47.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:47.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:48.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:48.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:48.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:48.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:49.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:49.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:49.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:49.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:50.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:50.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:50.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:50.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:51.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:51.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:51.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:51.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:51.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:51.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:52.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:52.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:52.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:52.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:53.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:53.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:53.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:53.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:54.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:54.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:54.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:54.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:55.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:55.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:55.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:55.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:55.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:58:55.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:55.829302+0000) 2019-10-18T02:58:56.326 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:58:56.320+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:56.325953+0000) 2019-10-18T02:58:56.484 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:58:56.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:58:56.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:58:56.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:56.672761+0000) 2019-10-18T02:58:56.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:56.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:56.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:56.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:56.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:58:56.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:58:56.718881+0000) 2019-10-18T02:58:57.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:57.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:57.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:57.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:58.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:58.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:58.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:58.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:58:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:59.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:58:59.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:58:59.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:00.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:00.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:00.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:00.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:01.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:01.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:01.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:01.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:01.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:01.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:02.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:02.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:02.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:02.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:03.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:03.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:03.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:03.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:04.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:04.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:04.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:04.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:05.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:05.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:05.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:05.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:05.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:59:05.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:05.829482+0000) 2019-10-18T02:59:06.326 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:59:06.320+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:06.326141+0000) 2019-10-18T02:59:06.486 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:06.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:06.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:59:06.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:06.672850+0000) 2019-10-18T02:59:06.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:59:06.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:06.718981+0000) 2019-10-18T02:59:06.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:06.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:06.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:06.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:07.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:07.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:07.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:07.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:08.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:08.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:08.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:08.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:09.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:09.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:09.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:09.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:10.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:10.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:10.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:10.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:11.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:11.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:11.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:11.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:11.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:11.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:12.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:12.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:12.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:12.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:13.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:13.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:13.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:13.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:14.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:14.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:14.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:14.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:15.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:59:15.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:15.829660+0000) 2019-10-18T02:59:15.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:15.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:15.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:15.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:16.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:59:16.324+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:16.326315+0000) 2019-10-18T02:59:16.488 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:16.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:16.672 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:59:16.667+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:16.672955+0000) 2019-10-18T02:59:16.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:59:16.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:16.719082+0000) 2019-10-18T02:59:16.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:16.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:16.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:16.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:17.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:17.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:17.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:17.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:18.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:18.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:18.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:18.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:19.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:19.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:19.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:19.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:20.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:20.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:20.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:20.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:21.489 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:21.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:21.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:21.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:21.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:21.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:22.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:22.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:22.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:22.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:23.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:23.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:23.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:23.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:24.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:24.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:24.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:24.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:25.829 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:59:25.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:25.829845+0000) 2019-10-18T02:59:25.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:25.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:25.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:25.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:26.326 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:59:26.324+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:26.326468+0000) 2019-10-18T02:59:26.489 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:26.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:26.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:59:26.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:26.673060+0000) 2019-10-18T02:59:26.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:59:26.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:26.719182+0000) 2019-10-18T02:59:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:26.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:26.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:27.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:27.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:27.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:27.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:28.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:28.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:28.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:28.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:29.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:29.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:29.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:29.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:30.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:30.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:30.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:30.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:31.491 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:31.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:31.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:31.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:31.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:31.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:32.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:32.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:32.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:32.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:33.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:33.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:33.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:33.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:34.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:34.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:34.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:34.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:35.830 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:59:35.824+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:35.830029+0000) 2019-10-18T02:59:35.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:35.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:35.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:35.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:36.326 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:59:36.324+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:36.326632+0000) 2019-10-18T02:59:36.491 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:36.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:36.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:59:36.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:36.673161+0000) 2019-10-18T02:59:36.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:59:36.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:36.719287+0000) 2019-10-18T02:59:36.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:36.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:36.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:36.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:37.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:37.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:37.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:37.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:38.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:38.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:38.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:38.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:39.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:39.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:39.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:40.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:40.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:41.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:41.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:41.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:41.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:41.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:41.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:42.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:42.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:42.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:42.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:43.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:43.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:43.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:43.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:44.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:44.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:44.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:44.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 324 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:45.830 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:59:45.828+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:45.830216+0000) 2019-10-18T02:59:45.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:45.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:45.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:45.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:46.326 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:59:46.324+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:46.326820+0000) 2019-10-18T02:59:46.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:46.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:46.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:59:46.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:46.673274+0000) 2019-10-18T02:59:46.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:59:46.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:46.719395+0000) 2019-10-18T02:59:46.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:46.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:46.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:46.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:47.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:47.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:47.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:47.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:48.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:48.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:48.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:48.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:49.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:49.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:49.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:49.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:51.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:51.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:51.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:51.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:51.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:51.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:52.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:51.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:52.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:51.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:52.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:52.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:52.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:52.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:53.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:53.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:53.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:53.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:54.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:54.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:54.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:54.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:55.830 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T02:59:55.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:55.830371+0000) 2019-10-18T02:59:55.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:55.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:55.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:55.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:56.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T02:59:56.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:56.326970+0000) 2019-10-18T02:59:56.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T02:59:56.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T02:59:56.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T02:59:56.666+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:56.673394+0000) 2019-10-18T02:59:56.721 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T02:59:56.714+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T01:59:56.719511+0000) 2019-10-18T02:59:56.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:56.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:56.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:56.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:57.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:57.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:57.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:57.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T02:59:59.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:59.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T02:59:59.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T02:59:59.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:00.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:00.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:00.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:00.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:01.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:01.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:01.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:01.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:01.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:01.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:02.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:02.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:02.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:02.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:03.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:03.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:03.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:03.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:03.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:03.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:04.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:04.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:04.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:04.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:05.830 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:00:05.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:05.830519+0000) 2019-10-18T03:00:05.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:05.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:05.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:05.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:06.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:00:06.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:06.327175+0000) 2019-10-18T03:00:06.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:06.489+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:06.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:00:06.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:06.673465+0000) 2019-10-18T03:00:06.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:00:06.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:06.719617+0000) 2019-10-18T03:00:06.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:06.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:06.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:06.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:07.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:07.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:07.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:07.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:08.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:08.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:08.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:08.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:09.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:09.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:09.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:10.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:10.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:10.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:10.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:11.497 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:11.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:11.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:11.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:11.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:11.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:12.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:12.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:12.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:12.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:13.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:13.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:13.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:13.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:14.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:14.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:14.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:14.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:15.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:15.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:15.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:15.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:15.831 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:00:15.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:15.830699+0000) 2019-10-18T03:00:16.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:00:16.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:16.327368+0000) 2019-10-18T03:00:16.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:16.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:16.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:00:16.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:16.673575+0000) 2019-10-18T03:00:16.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:00:16.713+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:16.719721+0000) 2019-10-18T03:00:16.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:16.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:16.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:16.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:17.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:17.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:17.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:17.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:18.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:18.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:18.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:18.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:19.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:19.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:19.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:19.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:20.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:20.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:20.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:20.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:21.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:21.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:21.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:21.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:21.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:21.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:22.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:22.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:22.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:22.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:23.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:23.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:23.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:23.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:24.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:24.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:24.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:24.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:25.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:25.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:25.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:25.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:25.830 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:00:25.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:25.830876+0000) 2019-10-18T03:00:26.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:00:26.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:26.327547+0000) 2019-10-18T03:00:26.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:26.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:26.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:00:26.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:26.673687+0000) 2019-10-18T03:00:26.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:26.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:26.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:26.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:26.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:00:26.717+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:26.719823+0000) 2019-10-18T03:00:27.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:27.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:27.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:27.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:28.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:28.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:28.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:28.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:29.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:29.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:29.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:29.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:30.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:30.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:30.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:30.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:31.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:31.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:31.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:31.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:31.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:31.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:32.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:32.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:32.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:32.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:33.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:33.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:33.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:33.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:34.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:34.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:34.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:34.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:35.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:35.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:35.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:35.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:35.831 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:00:35.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:35.831055+0000) 2019-10-18T03:00:36.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:00:36.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:36.327760+0000) 2019-10-18T03:00:36.502 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:36.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:36.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:00:36.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:36.673786+0000) 2019-10-18T03:00:36.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:36.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:36.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:36.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:36.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:00:36.717+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:36.719921+0000) 2019-10-18T03:00:37.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:37.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T02:50:01.377343+0000 (oldest deadline 2019-10-18T02:50:21.377343+0000) 2019-10-18T03:00:37.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:37.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:38.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:38.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:39.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:39.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:40.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:40.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:41.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:41.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:41.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:41.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:42.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:42.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:43.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:43.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:44.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:44.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:45.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:45.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:45.831 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:00:45.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:45.831242+0000) 2019-10-18T03:00:46.327 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:00:46.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:46.327939+0000) 2019-10-18T03:00:46.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:46.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:46.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:46.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:46.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:00:46.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:46.673888+0000) 2019-10-18T03:00:46.719 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:00:46.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:46.720018+0000) 2019-10-18T03:00:47.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:47.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:48.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:48.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:49.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:49.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:50.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:50.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:51.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:51.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:51.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:51.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:52.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:52.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:53.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:53.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:54.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:54.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:55.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:55.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:55.831 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:00:55.827+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:55.831424+0000) 2019-10-18T03:00:56.328 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:00:56.323+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:56.328129+0000) 2019-10-18T03:00:56.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:00:56.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:00:56.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:56.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:56.673 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:00:56.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:56.673988+0000) 2019-10-18T03:00:56.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:00:56.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:00:56.720112+0000) 2019-10-18T03:00:57.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:57.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:58.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:00:59.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:00:59.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:00.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:00.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:01.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:01.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:01.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:01.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:01.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:01.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:02.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:02.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:02.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:02.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:03.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:03.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:03.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:03.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:04.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:04.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:04.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:04.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:05.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:05.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:05.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:05.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:05.831 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:05.826+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:05.831609+0000) 2019-10-18T03:01:06.328 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:01:06.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:06.328319+0000) 2019-10-18T03:01:06.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:06.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:06.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:01:06.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:06.674096+0000) 2019-10-18T03:01:06.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:06.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:06.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:06.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:06.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:06.716+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:06.720209+0000) 2019-10-18T03:01:07.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:07.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:07.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:07.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:08.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:08.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:08.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:08.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:09.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:09.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:09.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:09.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:10.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:10.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:10.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:10.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:11.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:11.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:11.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:11.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:11.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:11.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:12.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:12.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:12.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:12.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:13.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:13.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:13.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:13.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:14.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:14.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:14.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:14.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:15.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:15.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:15.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:15.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:15.831 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:15.826+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:15.831809+0000) 2019-10-18T03:01:16.328 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:01:16.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:16.328499+0000) 2019-10-18T03:01:16.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:16.503+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:16.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:01:16.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:16.674211+0000) 2019-10-18T03:01:16.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:16.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:16.720309+0000) 2019-10-18T03:01:16.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:16.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:16.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:16.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:17.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:17.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:17.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:17.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:18.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:18.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:18.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:18.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:19.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:19.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:19.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:19.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:20.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:20.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:20.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:20.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:21.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:21.507+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:21.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:21.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:21.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:21.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:22.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:22.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:22.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:22.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:23.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:23.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:23.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:23.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:24.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:24.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:24.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:24.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:25.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:25.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:25.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:25.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:25.832 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:25.826+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:25.831981+0000) 2019-10-18T03:01:25.832 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:25.826+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:25.833109+0000) 2019-10-18T03:01:26.328 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:01:26.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:26.328685+0000) 2019-10-18T03:01:26.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:26.507+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:26.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:01:26.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:26.674319+0000) 2019-10-18T03:01:26.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:26.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:26.720410+0000) 2019-10-18T03:01:26.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:26.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:26.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:26.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:27.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:27.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:27.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:27.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:28.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:28.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:28.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:28.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:29.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:29.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:29.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:29.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:30.332 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:30.326+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:30.332451+0000) 2019-10-18T03:01:30.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:30.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:30.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:30.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:31.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:31.507+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:31.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:31.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:31.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:31.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:32.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:32.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:32.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:32.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:33.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:33.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:33.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:33.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:34.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:34.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:34.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:34.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:35.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:35.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:35.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:35.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:36.328 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:01:36.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:36.328873+0000) 2019-10-18T03:01:36.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:36.507+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:36.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:01:36.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:36.674436+0000) 2019-10-18T03:01:36.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:36.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:36.720513+0000) 2019-10-18T03:01:36.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:36.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:36.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:36.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:37.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:37.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:37.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:37.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:38.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:38.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:38.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:38.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:39.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:39.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:39.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:39.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:40.332 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:40.326+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:40.332632+0000) 2019-10-18T03:01:40.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:40.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:40.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:40.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:41.513 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:41.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:41.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:41.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:41.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:41.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:42.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:42.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:42.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:42.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:43.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:43.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:43.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:43.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:44.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:44.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:44.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:45.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:45.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:45.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:45.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:46.329 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:01:46.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:46.329054+0000) 2019-10-18T03:01:46.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:46.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:46.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:46.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:46.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:46.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:46.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:01:46.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:46.674540+0000) 2019-10-18T03:01:46.720 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:46.715+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:46.720608+0000) 2019-10-18T03:01:46.721 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:46.719+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:46.721750+0000) 2019-10-18T03:01:47.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:47.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:47.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:47.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:48.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:48.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:48.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:48.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:49.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:49.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:49.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:49.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:50.332 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:01:50.326+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:50.332825+0000) 2019-10-18T03:01:50.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:50.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:50.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:50.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:51.220 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:01:51.214+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:51.220797+0000) 2019-10-18T03:01:51.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:51.510+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:51.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:51.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:51.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:51.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:52.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:52.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:52.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:52.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:53.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:53.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:53.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:53.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:54.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:54.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:54.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:54.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:55.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:55.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:55.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:55.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:56.329 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:01:56.322+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:56.329236+0000) 2019-10-18T03:01:56.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:01:56.510+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:01:56.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:56.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:56.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:56.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:56.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:01:56.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:01:56.674650+0000) 2019-10-18T03:01:57.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:57.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:57.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:57.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:58.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:58.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:58.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:58.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:01:59.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:59.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:01:59.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:01:59.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:00.333 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:02:00.326+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:00.333010+0000) 2019-10-18T03:02:00.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:00.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:00.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:00.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:01.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:02:01.214+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:01.220894+0000) 2019-10-18T03:02:01.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:01.510+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:01.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:01.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:01.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:01.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:02.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:02.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:02.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:02.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:03.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:03.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:03.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:03.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:04.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:04.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:04.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:04.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:05.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:05.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:05.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:05.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:06.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:02:06.326+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:06.329420+0000) 2019-10-18T03:02:06.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:06.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:06.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:02:06.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:06.674766+0000) 2019-10-18T03:02:06.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:06.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:06.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:06.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:07.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:07.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:07.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:07.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:08.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:08.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:08.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:08.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:09.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:09.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:09.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:09.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:10.333 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:02:10.330+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:10.333194+0000) 2019-10-18T03:02:10.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:10.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:10.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:10.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:11.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:02:11.214+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:11.221119+0000) 2019-10-18T03:02:11.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:11.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:11.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:11.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:11.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:11.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:12.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:12.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:12.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:12.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:13.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:13.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:13.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:13.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:14.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:14.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:14.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:14.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:15.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:15.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:15.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:15.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:16.329 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:02:16.325+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:16.329604+0000) 2019-10-18T03:02:16.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:16.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:16.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:16.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:16.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:16.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:16.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:02:16.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:16.674887+0000) 2019-10-18T03:02:17.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:17.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:17.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:17.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:18.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:18.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:18.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:18.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:19.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:19.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:19.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:19.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:20.333 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:02:20.329+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:20.333379+0000) 2019-10-18T03:02:20.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:20.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:20.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:20.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:21.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:02:21.214+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:21.221246+0000) 2019-10-18T03:02:21.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:21.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:21.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:21.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:21.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:21.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:22.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:22.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:22.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:22.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:23.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:23.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:23.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:23.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:24.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:24.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:24.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:24.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:25.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:25.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:25.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:25.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:26.329 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:02:26.325+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:26.329781+0000) 2019-10-18T03:02:26.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:26.513+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:26.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:26.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:26.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:26.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:26.674 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:02:26.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:26.674992+0000) 2019-10-18T03:02:27.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:27.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:27.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:27.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:28.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:28.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:28.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:28.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:29.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:29.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:29.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:29.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:30.338 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:02:30.329+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:30.333559+0000) 2019-10-18T03:02:30.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:30.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:30.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:30.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:31.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:02:31.213+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:31.221347+0000) 2019-10-18T03:02:31.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:31.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:31.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:31.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:31.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:31.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:32.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:32.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:32.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:32.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:33.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:33.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:33.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:33.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:34.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:34.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:34.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:34.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:35.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:35.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:35.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:35.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:36.329 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:02:36.325+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:36.329963+0000) 2019-10-18T03:02:36.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:36.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:36.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:36.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:36.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:36.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:36.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:02:36.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:36.675100+0000) 2019-10-18T03:02:37.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:37.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:37.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:37.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:38.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:38.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:38.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:38.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:39.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:39.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:39.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:39.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:40.333 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:02:40.329+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:40.333776+0000) 2019-10-18T03:02:40.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:40.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:40.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:40.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:41.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:02:41.217+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:41.221453+0000) 2019-10-18T03:02:41.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:41.517+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:41.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:41.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:41.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:41.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:42.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:42.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:42.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:42.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:43.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:43.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:43.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:43.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:44.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:44.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:44.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:44.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:45.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:45.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:45.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:45.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:46.330 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:02:46.325+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:46.330157+0000) 2019-10-18T03:02:46.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:46.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:46.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:46.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:46.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:46.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:46.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:02:46.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:46.675217+0000) 2019-10-18T03:02:47.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:47.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:47.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:47.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:48.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:48.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:48.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:48.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:49.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:49.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:49.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:49.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:50.333 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:02:50.329+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:50.333959+0000) 2019-10-18T03:02:50.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:50.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:50.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:50.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:51.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:02:51.221+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:51.221549+0000) 2019-10-18T03:02:51.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:51.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:51.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:51.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:51.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:51.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:52.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:52.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:52.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:52.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:53.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:53.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:53.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:53.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:54.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:54.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:54.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:54.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:55.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:55.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:55.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:55.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:56.330 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:02:56.329+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:56.330336+0000) 2019-10-18T03:02:56.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:02:56.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:02:56.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:56.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:56.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:56.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:56.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:02:56.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:02:56.675338+0000) 2019-10-18T03:02:57.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:57.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:57.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:57.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:58.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:58.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:58.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:58.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:02:59.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:59.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:02:59.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:02:59.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:00.334 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:03:00.329+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:00.334140+0000) 2019-10-18T03:03:00.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:00.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:00.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:00.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:01.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:03:01.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:01.221651+0000) 2019-10-18T03:03:01.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:01.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:01.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:01.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:01.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:01.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:02.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:02.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:02.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:02.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:03.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:03.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:03.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:03.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:04.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:04.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:04.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:04.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:05.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:05.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:06.330 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:03:06.329+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:06.330482+0000) 2019-10-18T03:03:06.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:06.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:06.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:03:06.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:06.675449+0000) 2019-10-18T03:03:06.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:06.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:06.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:06.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:07.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:07.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:07.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:07.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:08.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:08.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:08.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:08.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:09.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:09.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:09.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:09.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:10.334 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:03:10.333+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:10.334332+0000) 2019-10-18T03:03:10.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:10.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:10.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:10.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:11.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:03:11.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:11.221753+0000) 2019-10-18T03:03:11.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:11.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:11.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:11.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:11.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:11.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:12.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:12.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:12.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:12.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:13.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:13.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:13.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:13.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:14.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:14.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:14.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:14.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:15.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:15.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:15.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:15.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:16.330 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:03:16.329+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:16.330658+0000) 2019-10-18T03:03:16.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:16.524+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:16.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:03:16.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:16.675557+0000) 2019-10-18T03:03:16.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:16.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:16.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:16.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:17.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:17.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:17.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:17.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:18.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:18.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:18.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:18.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:19.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:19.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:19.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:19.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:20.334 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:03:20.333+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:20.334493+0000) 2019-10-18T03:03:20.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:20.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:20.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:20.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:21.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:03:21.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:21.221850+0000) 2019-10-18T03:03:21.529 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:21.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:21.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:21.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:21.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:21.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:22.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:22.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:22.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:22.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:23.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:23.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:23.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:23.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:24.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:24.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:24.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:24.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:25.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:25.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:25.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:25.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:26.330 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:03:26.329+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:26.330832+0000) 2019-10-18T03:03:26.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:26.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:26.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:26.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:26.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:26.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:26.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:03:26.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:26.675670+0000) 2019-10-18T03:03:27.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:27.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:27.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:27.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:28.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:28.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:28.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:28.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:29.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:29.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:29.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:29.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:30.334 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:03:30.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:30.334684+0000) 2019-10-18T03:03:30.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:30.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:30.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:30.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:31.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:03:31.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:31.221947+0000) 2019-10-18T03:03:31.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:31.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:31.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:31.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:31.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:31.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:32.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:32.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:32.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:32.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:33.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:33.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:33.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:33.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:34.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:34.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:34.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:34.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:35.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:35.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:35.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:35.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:36.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:03:36.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:36.331030+0000) 2019-10-18T03:03:36.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:36.527+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:36.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:36.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:36.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:36.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:36.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:03:36.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:36.675794+0000) 2019-10-18T03:03:37.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:37.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:37.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:37.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:38.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:38.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:38.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:38.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:39.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:39.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:39.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:39.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:40.336 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:03:40.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:40.334864+0000) 2019-10-18T03:03:40.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:40.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:40.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:40.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:41.221 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:03:41.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:41.222046+0000) 2019-10-18T03:03:41.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:41.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:41.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:41.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:41.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:41.531+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:42.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:42.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:42.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:42.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:43.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:43.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:43.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:43.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:44.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:44.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:44.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:44.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:45.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:45.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:45.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:45.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:46.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:03:46.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:46.331214+0000) 2019-10-18T03:03:46.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:46.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:46.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:46.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:46.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:46.531+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:46.675 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:03:46.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:46.675898+0000) 2019-10-18T03:03:47.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:47.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:47.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:47.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:48.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:48.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:48.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:48.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:49.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:49.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:49.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:49.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:50.335 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:03:50.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:50.335051+0000) 2019-10-18T03:03:50.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:50.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:50.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:50.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:51.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:03:51.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:51.222143+0000) 2019-10-18T03:03:51.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:51.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:51.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:51.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:51.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:51.531+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:52.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:52.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:52.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:52.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:53.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:53.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:53.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:53.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:54.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:54.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:54.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:54.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:55.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:55.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:55.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:55.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:56.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:03:56.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:56.331407+0000) 2019-10-18T03:03:56.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:03:56.531+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:03:56.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:56.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:56.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:56.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:56.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:03:56.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:03:56.676000+0000) 2019-10-18T03:03:57.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:57.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:57.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:57.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:58.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:58.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:58.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:58.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:03:59.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:59.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:03:59.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:03:59.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:00.335 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:04:00.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:00.335233+0000) 2019-10-18T03:04:00.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:00.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:00.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:00.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:01.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:04:01.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:01.222247+0000) 2019-10-18T03:04:01.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:01.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:01.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:01.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:01.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:01.531+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:02.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:02.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:02.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:02.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:03.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:03.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:03.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:03.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:04.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:04.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:04.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:04.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:05.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:05.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:05.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:05.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:06.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:04:06.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:06.331587+0000) 2019-10-18T03:04:06.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:06.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:06.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:06.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:06.537 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:06.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:06.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:04:06.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:06.676117+0000) 2019-10-18T03:04:07.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:07.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:07.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:07.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:08.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:08.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:08.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:08.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:09.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:09.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:09.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:09.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:10.335 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:04:10.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:10.335418+0000) 2019-10-18T03:04:10.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:10.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:10.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:10.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:11.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:04:11.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:11.222349+0000) 2019-10-18T03:04:11.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:11.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:11.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:11.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:11.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:11.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:12.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:12.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:12.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:12.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:13.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:13.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:13.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:13.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:14.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:14.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:14.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:14.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:15.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:15.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:15.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:15.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:16.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:04:16.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:16.331764+0000) 2019-10-18T03:04:16.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:16.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:16.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:16.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:16.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:16.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:16.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:04:16.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:16.676237+0000) 2019-10-18T03:04:17.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:17.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:17.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:17.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:18.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:18.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:18.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:18.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:19.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:19.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:19.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:19.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:20.335 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:04:20.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:20.335596+0000) 2019-10-18T03:04:20.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:20.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:20.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:20.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:21.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:04:21.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:21.222446+0000) 2019-10-18T03:04:21.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:21.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:21.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:21.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:21.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:21.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:22.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:22.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:22.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:22.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:23.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:23.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:23.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:23.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:24.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:24.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:24.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:24.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:25.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:25.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:25.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:25.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:26.331 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:04:26.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:26.331946+0000) 2019-10-18T03:04:26.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:26.534+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:26.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:26.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:26.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:26.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:26.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:04:26.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:26.676337+0000) 2019-10-18T03:04:27.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:27.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:27.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:27.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:28.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:28.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:28.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:28.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:29.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:29.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:29.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:29.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:30.335 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:04:30.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:30.335782+0000) 2019-10-18T03:04:30.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:30.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:30.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:30.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:31.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:04:31.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:31.222549+0000) 2019-10-18T03:04:31.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:31.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:31.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:31.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:31.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:31.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:32.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:32.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:32.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:32.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:33.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:33.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:33.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:33.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:34.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:34.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:34.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:34.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:35.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:35.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:35.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:35.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:36.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:04:36.328+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:36.332131+0000) 2019-10-18T03:04:36.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:36.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:36.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:36.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:36.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:36.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:36.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:04:36.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:36.676441+0000) 2019-10-18T03:04:37.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:37.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:37.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:37.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:38.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:38.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:38.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:38.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:39.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:39.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:39.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:39.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:40.336 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:04:40.332+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:40.335973+0000) 2019-10-18T03:04:40.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:40.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:40.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:40.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:41.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:04:41.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:41.222648+0000) 2019-10-18T03:04:41.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:41.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:41.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:41.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:41.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:41.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:42.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:42.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:42.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:42.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:43.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:43.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:43.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:43.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:44.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:44.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:44.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:44.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:45.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:45.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:45.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:45.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:46.332 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:04:46.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:46.332312+0000) 2019-10-18T03:04:46.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:46.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:46.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:04:46.670+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:46.676547+0000) 2019-10-18T03:04:46.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:46.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:46.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:46.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:47.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:47.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:47.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:47.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:48.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:48.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:48.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:48.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:49.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:49.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:49.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:49.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:50.336 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:04:50.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:50.336183+0000) 2019-10-18T03:04:50.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:50.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:50.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:50.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:51.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:04:51.217+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:51.222744+0000) 2019-10-18T03:04:51.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:51.537+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:51.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:51.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:51.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:51.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:52.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:52.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:52.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:52.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:53.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:53.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:53.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:53.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:54.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:54.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:54.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:54.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:55.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:55.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:55.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:55.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:56.333 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:04:56.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:56.332519+0000) 2019-10-18T03:04:56.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:04:56.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:04:56.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:56.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:56.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:56.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:56.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:04:56.669+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:04:56.676649+0000) 2019-10-18T03:04:57.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:57.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:57.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:57.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:58.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:58.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:58.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:58.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:04:59.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:59.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:04:59.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:04:59.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:00.336 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:05:00.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:00.336373+0000) 2019-10-18T03:05:00.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:00.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:00.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:00.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:01.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:05:01.217+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:01.222851+0000) 2019-10-18T03:05:01.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:01.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:01.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:01.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:01.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:01.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:02.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:02.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:02.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:02.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:03.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:03.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:03.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:03.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:04.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:04.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:04.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:04.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:05.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:05.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:05.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:05.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:06.332 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:05:06.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:06.332703+0000) 2019-10-18T03:05:06.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:06.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:06.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:05:06.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:06.676752+0000) 2019-10-18T03:05:06.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:06.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:06.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:06.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:07.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:07.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:07.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:07.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:08.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:08.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:08.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:08.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:09.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:09.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:09.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:09.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:10.336 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:05:10.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:10.336570+0000) 2019-10-18T03:05:10.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:10.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:10.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:10.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:11.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:05:11.221+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:11.222957+0000) 2019-10-18T03:05:11.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:11.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:11.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:11.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:11.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:11.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:12.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:12.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:12.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:12.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:13.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:13.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:13.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:13.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:14.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:14.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:14.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:14.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:15.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:15.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:15.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:15.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:16.332 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:05:16.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:16.332883+0000) 2019-10-18T03:05:16.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:16.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:16.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:05:16.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:16.676857+0000) 2019-10-18T03:05:16.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:16.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:16.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:16.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:17.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:17.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:17.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:17.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:18.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:18.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:18.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:18.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:19.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:19.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:19.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:19.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:20.337 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:05:20.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:20.336757+0000) 2019-10-18T03:05:20.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:20.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:20.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:20.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:21.222 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:05:21.221+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:21.223054+0000) 2019-10-18T03:05:21.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:21.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:21.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:21.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:21.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:21.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:22.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:22.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:22.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:22.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:23.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:23.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:23.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:23.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:24.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:24.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:24.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:24.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:25.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:25.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:25.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:25.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:26.332 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:05:26.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:26.333037+0000) 2019-10-18T03:05:26.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:26.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:26.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:05:26.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:26.676955+0000) 2019-10-18T03:05:26.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:26.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:26.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:26.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:27.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:27.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:27.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:27.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:28.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:28.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:28.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:28.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:29.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:29.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:30.336 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:05:30.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:30.336937+0000) 2019-10-18T03:05:30.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:30.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:30.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:30.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:31.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:05:31.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:31.223153+0000) 2019-10-18T03:05:31.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:31.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:31.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:31.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:31.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:31.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:32.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:32.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:32.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:32.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:33.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:33.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:33.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:33.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:34.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:34.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:34.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:34.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:35.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:35.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:35.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:35.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:36.333 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:05:36.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:36.333206+0000) 2019-10-18T03:05:36.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:36.548+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:36.676 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:05:36.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:36.677063+0000) 2019-10-18T03:05:36.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:36.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:36.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:36.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:37.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:37.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:37.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:37.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:38.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:38.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:38.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:38.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:39.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:39.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:39.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:39.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:40.337 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:05:40.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:40.337144+0000) 2019-10-18T03:05:40.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:40.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:40.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:40.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:41.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:05:41.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:41.223255+0000) 2019-10-18T03:05:41.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:41.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:41.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:41.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:41.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:41.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:42.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:42.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:42.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:42.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:43.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:43.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:43.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:43.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:44.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:44.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:44.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:44.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:45.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:45.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:45.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:45.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:46.333 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:05:46.327+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:46.333404+0000) 2019-10-18T03:05:46.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:46.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:46.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:05:46.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:46.677158+0000) 2019-10-18T03:05:46.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:46.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:46.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:46.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:47.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:47.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:47.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:47.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:48.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:48.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:48.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:48.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:49.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:49.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:49.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:49.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:50.337 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:05:50.331+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:50.337338+0000) 2019-10-18T03:05:50.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:50.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:50.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:50.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:51.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:05:51.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:51.223354+0000) 2019-10-18T03:05:51.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:51.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:51.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:51.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:51.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:51.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:52.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:52.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:52.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:52.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:53.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:53.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:53.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:53.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:54.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:54.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:54.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:54.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:55.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:55.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:55.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:55.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:56.333 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:05:56.326+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:56.333583+0000) 2019-10-18T03:05:56.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:05:56.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:05:56.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:56.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:56.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:56.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:56.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:05:56.672+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:05:56.677275+0000) 2019-10-18T03:05:57.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:57.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:57.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:57.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:58.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:58.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:58.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:58.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:05:59.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:59.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:05:59.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:05:59.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:00.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:00.330+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:00.337500+0000) 2019-10-18T03:06:00.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:00.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:00.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:00.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:01.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:06:01.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:01.223458+0000) 2019-10-18T03:06:01.556 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:01.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:01.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:01.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:01.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:01.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:02.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:02.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:02.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:02.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:03.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:03.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:03.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:03.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:04.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:04.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:04.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:04.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:05.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:05.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:05.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:05.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:06.333 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:06:06.326+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:06.333770+0000) 2019-10-18T03:06:06.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:06.551+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:06.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:06:06.671+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:06.677384+0000) 2019-10-18T03:06:06.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:06.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:06.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:06.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:07.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:07.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:07.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:07.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:08.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:08.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:08.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:08.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:09.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:09.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:09.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:09.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:10.338 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:10.330+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:10.337628+0000) 2019-10-18T03:06:10.338 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:10.334+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:10.338757+0000) 2019-10-18T03:06:10.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:10.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:10.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:10.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:11.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:06:11.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:11.223559+0000) 2019-10-18T03:06:11.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:11.555+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:11.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:11.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:11.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:11.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:12.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:12.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:12.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:13.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:13.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:14.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:14.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:14.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:14.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:14.837 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:14.830+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:14.837849+0000) 2019-10-18T03:06:15.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:15.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:15.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:15.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:16.333 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:06:16.326+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:16.333910+0000) 2019-10-18T03:06:16.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:16.555+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:16.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:06:16.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:16.677491+0000) 2019-10-18T03:06:16.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:16.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:16.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:16.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:17.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:17.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:17.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:17.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:18.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:18.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:18.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:18.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:19.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:19.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:19.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:19.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:20.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:20.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:21.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:06:21.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:21.223663+0000) 2019-10-18T03:06:21.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:21.555+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:21.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:21.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:21.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:21.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:22.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:22.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:22.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:22.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:23.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:23.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:23.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:23.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:24.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:24.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:24.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:24.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:24.837 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:24.830+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:24.837964+0000) 2019-10-18T03:06:25.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:25.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:25.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:25.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:26.334 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:06:26.326+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:26.334101+0000) 2019-10-18T03:06:26.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:26.555+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:26.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:06:26.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:26.677593+0000) 2019-10-18T03:06:26.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:26.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:26.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:26.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:27.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:27.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:28.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:28.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:28.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:28.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:29.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:29.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:29.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:29.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:30.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:30.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:30.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:30.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:31.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:06:31.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:31.223767+0000) 2019-10-18T03:06:31.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:31.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:31.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:31.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:31.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:31.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:32.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:32.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:32.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:32.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:33.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:33.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:33.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:33.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:34.841 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:34.830+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:34.838082+0000) 2019-10-18T03:06:34.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:34.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:34.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:34.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:35.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:35.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:35.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:35.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:36.334 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:06:36.330+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:36.334279+0000) 2019-10-18T03:06:36.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:36.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:36.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:06:36.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:36.677704+0000) 2019-10-18T03:06:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:36.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:36.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:37.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:37.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:37.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:37.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:38.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:38.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:38.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:38.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:39.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:39.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:39.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:39.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:40.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:40.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:40.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:40.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:41.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:06:41.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:41.223871+0000) 2019-10-18T03:06:41.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:41.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:41.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:41.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:41.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:41.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:42.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:42.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:42.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:42.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:43.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:43.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:43.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:43.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:44.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:44.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:44.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:44.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:44.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:44.834+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:44.838209+0000) 2019-10-18T03:06:45.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:45.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:45.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:45.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:46.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:06:46.330+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:46.334481+0000) 2019-10-18T03:06:46.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:46.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:46.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:06:46.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:46.677825+0000) 2019-10-18T03:06:46.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:46.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:46.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:46.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:47.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:47.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:47.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:47.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:48.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:48.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:48.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:48.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:49.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:49.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:49.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:49.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:50.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:50.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:50.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:50.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:51.223 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:06:51.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:51.223982+0000) 2019-10-18T03:06:51.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:51.558+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:51.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:51.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:51.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:51.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:52.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:52.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:52.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:52.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:53.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:53.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:53.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:53.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:54.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:54.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:54.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:06:54.838+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:54.838306+0000) 2019-10-18T03:06:55.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:55.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:55.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:55.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:56.334 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:06:56.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:56.334666+0000) 2019-10-18T03:06:56.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:06:56.562+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:06:56.677 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:06:56.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:06:56.677954+0000) 2019-10-18T03:06:56.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:56.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:56.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:56.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:57.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:57.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:57.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:57.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:58.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:58.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:58.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:58.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:06:59.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:59.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:06:59.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:06:59.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:00.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:00.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:00.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:00.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:01.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:07:01.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:01.224082+0000) 2019-10-18T03:07:01.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:01.562+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:01.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:01.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:01.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:01.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:02.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:02.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:02.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:02.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:03.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:03.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:03.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:03.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:04.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:04.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:04.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:04.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:04.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:07:04.838+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:04.838408+0000) 2019-10-18T03:07:05.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:05.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:05.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:05.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:06.334 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:07:06.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:06.334844+0000) 2019-10-18T03:07:06.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:06.562+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:06.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:07:06.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:06.678078+0000) 2019-10-18T03:07:06.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:06.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:06.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:06.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:07.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:07.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:07.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:07.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:08.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:08.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:08.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:08.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:09.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:09.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:09.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:09.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:10.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:10.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:10.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:10.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:11.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:07:11.217+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:11.224186+0000) 2019-10-18T03:07:11.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:11.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:11.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:11.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:11.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:11.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:12.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:12.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:12.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:12.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:13.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:13.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:13.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:13.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:14.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:07:14.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:14.838505+0000) 2019-10-18T03:07:14.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:14.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:14.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:14.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:15.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:15.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:15.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:15.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:16.334 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:07:16.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:16.335022+0000) 2019-10-18T03:07:16.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:16.561+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:16.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:07:16.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:16.678202+0000) 2019-10-18T03:07:16.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:16.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:16.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:16.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:17.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:17.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:17.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:17.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:18.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:18.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:18.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:18.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:19.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:19.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:19.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:19.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:20.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:20.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:20.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:20.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:21.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:07:21.217+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:21.224283+0000) 2019-10-18T03:07:21.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:21.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:21.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:21.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:21.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:21.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:22.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:22.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:22.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:22.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:23.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:23.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:23.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:23.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:24.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:07:24.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:24.838601+0000) 2019-10-18T03:07:24.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:24.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:24.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:24.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:25.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:25.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:25.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:25.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:26.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:07:26.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:26.335204+0000) 2019-10-18T03:07:26.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:26.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:26.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:07:26.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:26.678316+0000) 2019-10-18T03:07:27.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:27.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:27.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:27.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:28.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:28.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:28.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:28.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:29.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:29.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:29.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:29.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:30.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:30.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:30.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:30.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:31.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:31.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:31.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:31.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:31.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:07:31.221+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:31.224387+0000) 2019-10-18T03:07:31.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:31.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:32.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:32.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:32.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:32.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:33.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:33.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:33.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:33.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:34.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:34.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:34.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:34.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:34.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:07:34.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:34.838711+0000) 2019-10-18T03:07:35.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:35.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:35.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:35.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:36.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:36.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:36.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:36.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:36.337 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:07:36.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:36.335417+0000) 2019-10-18T03:07:36.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:36.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:36.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:07:36.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:36.678440+0000) 2019-10-18T03:07:37.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:37.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:37.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:37.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:38.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:38.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:38.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:38.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:39.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:39.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:39.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:39.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:40.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:40.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:40.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:40.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:41.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:41.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:41.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:41.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:41.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:07:41.221+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:41.224485+0000) 2019-10-18T03:07:41.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:41.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:42.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:42.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:42.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:42.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:43.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:43.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:43.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:43.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:44.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:44.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:44.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:44.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:44.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:07:44.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:44.838824+0000) 2019-10-18T03:07:45.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:45.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:45.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:45.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:46.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:46.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:46.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:46.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:46.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:07:46.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:46.335597+0000) 2019-10-18T03:07:46.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:46.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:46.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:07:46.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:46.678548+0000) 2019-10-18T03:07:47.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:47.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:47.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:47.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:48.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:48.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:48.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:48.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:49.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:49.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:49.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:49.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:50.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:50.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:50.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:50.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:51.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:51.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:51.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:51.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:51.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:07:51.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:51.224583+0000) 2019-10-18T03:07:51.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:51.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:52.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:52.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:52.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:52.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:53.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:53.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:53.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:53.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:54.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:54.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:54.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:54.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:54.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:07:54.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:54.838938+0000) 2019-10-18T03:07:55.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:55.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:55.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:55.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:56.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:56.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:56.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:56.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:56.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:07:56.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:56.335773+0000) 2019-10-18T03:07:56.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:07:56.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:07:56.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:07:56.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:07:56.678671+0000) 2019-10-18T03:07:57.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:57.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:57.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:57.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:58.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:58.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:58.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:58.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:07:59.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:59.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:07:59.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:07:59.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:00.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:00.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:00.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:00.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:00.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:00.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:00.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:00.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:01.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:08:01.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:01.224687+0000) 2019-10-18T03:08:01.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:01.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:01.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:01.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:01.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:01.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:02.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:02.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:02.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:02.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:03.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:03.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:03.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:03.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:04.838 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:08:04.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:04.839047+0000) 2019-10-18T03:08:04.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:04.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:04.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:04.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:05.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:05.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:05.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:05.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:06.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:08:06.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:06.335945+0000) 2019-10-18T03:08:06.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:06.572+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:06.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:08:06.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:06.678789+0000) 2019-10-18T03:08:06.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:06.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:06.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:06.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:07.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:07.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:07.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:07.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:08.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:08.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:08.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:08.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:10.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:10.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:10.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:10.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:10.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:10.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:10.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:10.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:11.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:08:11.220+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:11.224792+0000) 2019-10-18T03:08:11.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:11.576+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:11.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:11.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:11.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:11.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:12.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:12.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:12.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:12.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:13.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:13.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:13.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:13.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:14.841 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:08:14.837+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:14.839158+0000) 2019-10-18T03:08:14.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:14.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:14.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:14.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:15.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:15.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:15.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:15.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:16.335 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:08:16.333+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:16.336160+0000) 2019-10-18T03:08:16.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:16.576+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:16.678 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:08:16.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:16.678920+0000) 2019-10-18T03:08:16.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:16.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:16.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:16.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:17.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:17.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:17.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:17.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:18.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:18.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:19.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:19.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:19.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:19.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:21.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:20.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:21.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:20.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:21.226 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:08:21.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:21.224888+0000) 2019-10-18T03:08:21.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:21.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:21.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:21.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:22.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:22.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:22.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:22.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:23.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:23.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:23.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:23.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:24.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:08:24.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:24.839271+0000) 2019-10-18T03:08:24.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:24.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:24.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:24.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:25.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:25.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:25.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:25.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:26.336 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:08:26.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:26.336344+0000) 2019-10-18T03:08:26.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:26.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:26.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:08:26.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:26.679039+0000) 2019-10-18T03:08:26.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:26.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:26.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:26.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:27.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:27.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:27.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:28.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:28.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:28.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:28.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:29.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:29.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:29.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:29.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:30.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:30.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:30.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:30.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:31.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:08:31.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:31.224990+0000) 2019-10-18T03:08:31.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:31.575+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:31.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:31.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:31.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:31.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:32.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:32.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:32.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:32.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:33.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:33.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:34.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:08:34.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:34.839384+0000) 2019-10-18T03:08:34.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:34.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:34.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:34.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:35.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:35.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:35.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:35.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:36.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:08:36.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:36.336520+0000) 2019-10-18T03:08:36.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:36.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:36.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:08:36.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:36.679165+0000) 2019-10-18T03:08:36.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:36.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:36.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:36.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:37.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:37.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:37.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:37.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:38.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:38.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:38.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:38.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:40.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:40.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:40.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:40.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:40.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:40.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:40.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:40.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:41.225 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:08:41.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:41.225087+0000) 2019-10-18T03:08:41.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:41.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:42.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:42.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:42.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:42.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:43.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:43.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:43.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:43.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:44.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:44.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:44.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:44.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:44.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:08:44.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:44.839499+0000) 2019-10-18T03:08:45.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:45.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:45.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:45.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:46.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:46.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:46.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:46.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:46.336 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:08:46.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:46.336706+0000) 2019-10-18T03:08:46.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:46.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:46.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:08:46.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:46.679285+0000) 2019-10-18T03:08:47.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:47.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:47.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:47.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:48.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:48.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:48.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:48.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:49.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:49.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:49.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:49.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:50.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:50.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:50.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:50.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:51.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:51.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:51.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:51.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:51.224 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:08:51.219+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:51.225184+0000) 2019-10-18T03:08:51.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:51.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:52.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:52.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:52.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:52.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:53.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:53.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:53.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:53.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:54.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:54.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:54.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:54.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:54.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:08:54.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:54.839611+0000) 2019-10-18T03:08:55.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:55.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:55.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:55.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:56.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:56.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:56.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:56.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:56.336 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:08:56.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:56.336887+0000) 2019-10-18T03:08:56.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:08:56.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:08:56.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:08:56.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:08:56.679413+0000) 2019-10-18T03:08:57.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:57.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:57.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:57.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:58.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:58.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:58.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:58.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:08:59.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:59.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:08:59.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:08:59.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:00.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:00.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:00.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:00.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:01.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:01.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:01.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:01.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:01.225 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:01.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:01.225284+0000) 2019-10-18T03:09:01.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:01.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:02.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:02.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:02.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:02.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:03.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:03.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:03.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:03.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:04.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:04.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:04.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:04.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:04.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:04.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:04.839724+0000) 2019-10-18T03:09:05.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:05.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:05.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:05.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:06.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:06.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:06.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:06.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:06.336 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:09:06.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:06.337068+0000) 2019-10-18T03:09:06.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:06.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:06.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:09:06.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:06.679543+0000) 2019-10-18T03:09:07.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:07.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:07.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:07.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:08.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:08.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:08.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:08.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:09.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:09.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:09.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:09.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:10.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:10.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:10.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:10.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:11.225 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:11.218+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:11.225379+0000) 2019-10-18T03:09:11.226 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:11.222+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:11.226694+0000) 2019-10-18T03:09:11.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:11.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:11.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:11.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:11.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:11.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:12.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:12.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:12.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:12.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:13.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:13.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:13.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:13.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:14.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:14.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:14.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:14.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:14.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:14.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:14.839829+0000) 2019-10-18T03:09:15.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:15.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:15.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:15.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:15.725 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:15.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:15.725599+0000) 2019-10-18T03:09:16.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:16.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:16.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:16.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:16.337 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:09:16.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:16.337248+0000) 2019-10-18T03:09:16.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:16.582+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:16.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:09:16.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:16.679671+0000) 2019-10-18T03:09:17.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:17.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:17.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:17.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:18.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:18.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:18.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:18.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:19.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:19.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:19.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:19.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:20.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:20.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:20.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:20.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:21.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:21.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:21.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:21.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:21.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:21.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:22.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:22.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:22.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:22.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:23.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:23.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:23.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:23.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:24.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:24.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:24.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:24.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:24.839 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:24.836+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:24.839935+0000) 2019-10-18T03:09:24.840 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:24.836+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:24.841059+0000) 2019-10-18T03:09:25.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:25.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:25.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:25.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:25.725 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:25.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:25.725692+0000) 2019-10-18T03:09:26.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:26.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:26.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:26.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:26.337 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:09:26.332+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:26.337430+0000) 2019-10-18T03:09:26.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:26.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:26.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:09:26.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:26.679781+0000) 2019-10-18T03:09:27.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:27.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:27.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:27.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:28.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:28.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:28.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:28.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:29.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:29.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:29.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:29.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:29.339 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:29.336+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:29.340146+0000) 2019-10-18T03:09:30.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:30.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:30.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:30.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:31.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:31.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:31.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:31.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:31.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:31.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:32.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:32.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:32.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:32.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:33.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:33.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:33.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:33.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:34.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:34.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:34.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:34.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:35.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:35.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:35.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:35.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:35.725 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:35.721+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:35.725788+0000) 2019-10-18T03:09:36.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:36.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:36.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:36.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:36.337 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:09:36.331+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:36.337611+0000) 2019-10-18T03:09:36.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:36.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:36.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:09:36.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:36.679909+0000) 2019-10-18T03:09:37.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:37.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:37.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:37.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:38.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:38.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:38.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:38.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:39.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:39.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:39.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:39.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:39.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:39.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:39.340259+0000) 2019-10-18T03:09:40.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:40.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:40.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:40.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:41.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:41.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:41.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:41.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:41.593 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:41.585+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:42.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:42.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:42.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:42.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:43.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:43.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:43.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:43.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:44.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:44.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:44.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:44.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:45.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:45.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:45.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:45.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:45.725 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:45.721+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:45.725885+0000) 2019-10-18T03:09:46.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:46.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:46.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:46.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:46.337 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:09:46.331+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:46.337789+0000) 2019-10-18T03:09:46.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:46.589+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:46.679 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:09:46.673+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:46.680029+0000) 2019-10-18T03:09:47.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:47.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:47.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:47.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:48.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:48.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:48.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:48.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:49.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:49.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:49.340371+0000) 2019-10-18T03:09:49.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:49.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:49.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:49.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:50.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:50.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:50.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:50.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:51.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:51.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:51.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:51.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:51.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:51.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:52.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:52.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:52.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:52.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:53.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:53.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:53.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:53.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:54.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:54.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:54.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:54.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:55.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:55.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:55.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:55.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:55.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:09:55.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:55.725985+0000) 2019-10-18T03:09:56.337 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:09:56.331+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:56.337941+0000) 2019-10-18T03:09:56.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:56.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:56.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:56.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:56.595 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:09:56.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:09:56.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:09:56.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:56.680156+0000) 2019-10-18T03:09:57.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:57.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:57.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:57.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:58.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:58.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:58.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:58.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:09:59.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:09:59.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:09:59.340482+0000) 2019-10-18T03:09:59.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:59.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:09:59.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:09:59.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:00.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:00.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:00.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:00.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:01.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:01.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:01.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:01.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:01.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:01.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:02.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:02.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:02.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:02.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:03.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:03.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:03.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:03.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:04.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:04.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:04.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:04.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:05.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:05.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:05.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:05.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:05.725 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:10:05.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:05.726076+0000) 2019-10-18T03:10:06.338 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:10:06.331+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:06.338158+0000) 2019-10-18T03:10:06.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:06.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:06.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:06.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:06.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:06.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:06.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:10:06.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:06.680277+0000) 2019-10-18T03:10:07.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:07.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:07.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:07.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:08.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:08.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:08.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:08.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:09.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:10:09.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:09.340581+0000) 2019-10-18T03:10:09.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:09.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:09.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:09.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:10.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:10.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:10.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:10.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:11.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:11.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:11.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:11.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:11.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:11.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:12.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:12.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:12.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:12.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:13.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:13.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:13.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:13.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:14.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:14.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:14.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:14.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:15.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:15.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:15.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:15.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:15.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:10:15.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:15.726166+0000) 2019-10-18T03:10:16.340 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:10:16.335+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:16.338369+0000) 2019-10-18T03:10:16.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:16.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:16.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:16.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:16.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:16.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:16.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:10:16.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:16.680408+0000) 2019-10-18T03:10:17.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:17.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:17.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:17.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:18.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:18.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:18.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:18.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:19.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:10:19.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:19.340695+0000) 2019-10-18T03:10:19.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:19.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:19.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:19.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:20.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:20.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:20.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:20.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:21.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:21.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:21.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:21.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:21.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:21.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:22.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:22.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:22.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:22.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:23.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:23.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:23.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:23.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:24.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:24.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:24.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:24.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:25.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:25.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:25.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:25.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:25.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:10:25.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:25.726264+0000) 2019-10-18T03:10:26.338 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:10:26.335+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:26.338547+0000) 2019-10-18T03:10:26.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:26.596+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:26.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:10:26.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:26.680536+0000) 2019-10-18T03:10:26.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:26.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:26.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:26.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:27.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:27.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:27.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:27.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:28.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:28.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:28.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:29.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:10:29.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:29.340806+0000) 2019-10-18T03:10:29.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:29.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:29.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:29.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:30.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:30.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:30.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:30.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:31.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:31.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:31.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:31.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:31.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:31.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:32.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:32.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:32.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:32.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:33.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:33.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:33.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:33.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:34.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:34.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:34.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:34.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:35.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:35.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:35.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:35.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:35.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:10:35.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:35.726356+0000) 2019-10-18T03:10:36.338 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:10:36.335+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:36.338723+0000) 2019-10-18T03:10:36.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:36.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:36.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:36.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:36.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:36.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:36.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:10:36.676+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:36.680678+0000) 2019-10-18T03:10:37.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:37.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:37.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:37.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:38.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:38.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:38.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:38.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:39.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:10:39.335+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:39.340920+0000) 2019-10-18T03:10:39.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:39.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:39.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:39.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:40.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:40.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:40.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:40.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:41.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:41.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:41.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:41.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:41.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:41.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:42.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:42.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:42.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:42.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:43.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:43.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:43.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:43.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:44.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:44.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:44.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:44.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:45.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:45.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:45.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:45.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:45.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:10:45.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:45.726450+0000) 2019-10-18T03:10:46.338 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:10:46.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:46.338903+0000) 2019-10-18T03:10:46.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:46.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:46.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:46.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:46.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:46.599+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:46.680 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:10:46.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:46.680799+0000) 2019-10-18T03:10:47.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:47.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:47.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:47.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:48.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:48.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:48.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:48.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:49.340 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:10:49.334+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:49.341034+0000) 2019-10-18T03:10:49.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:49.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:49.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:49.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:50.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:50.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:50.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:50.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:51.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:51.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:51.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:51.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:51.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:51.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:52.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:52.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:52.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:52.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:53.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:53.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:53.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:53.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:54.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:54.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:54.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:54.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:55.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:55.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:55.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:55.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:55.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:10:55.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:55.726548+0000) 2019-10-18T03:10:56.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:10:56.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:56.339084+0000) 2019-10-18T03:10:56.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:56.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:56.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:56.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:56.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:10:56.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:10:56.691 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:10:56.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:56.680931+0000) 2019-10-18T03:10:57.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:57.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:57.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:57.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:58.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:58.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:58.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:58.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:10:59.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:10:59.334+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:10:59.341158+0000) 2019-10-18T03:10:59.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:59.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:10:59.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:10:59.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:00.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:00.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:00.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:00.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:01.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:01.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:01.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:01.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:01.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:01.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:02.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:02.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:02.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:02.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:03.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:03.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:03.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:03.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:04.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:04.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:04.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:04.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:05.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:05.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:05.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:05.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:05.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:11:05.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:05.726643+0000) 2019-10-18T03:11:06.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:11:06.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:06.339266+0000) 2019-10-18T03:11:06.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:06.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:06.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:06.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:06.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:06.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:06.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:11:06.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:06.681056+0000) 2019-10-18T03:11:07.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:07.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:07.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:07.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:08.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:08.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:08.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:08.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:09.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:11:09.334+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:09.341255+0000) 2019-10-18T03:11:09.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:09.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:00:40.910853+0000 (oldest deadline 2019-10-18T03:01:00.910853+0000) 2019-10-18T03:11:09.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:09.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:10.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:10.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:11.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:11.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:11.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:11.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:12.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:12.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:13.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:13.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:14.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:14.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:15.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:15.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:15.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:11:15.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:15.726741+0000) 2019-10-18T03:11:16.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:11:16.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:16.339448+0000) 2019-10-18T03:11:16.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:16.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:16.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:16.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:16.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:11:16.675+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:16.681186+0000) 2019-10-18T03:11:17.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:17.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:18.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:18.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:19.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:11:19.334+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:19.341361+0000) 2019-10-18T03:11:19.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:19.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:20.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:20.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:21.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:21.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:21.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:21.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:22.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:22.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:23.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:23.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:24.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:24.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:25.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:25.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:25.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:11:25.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:25.726840+0000) 2019-10-18T03:11:26.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:11:26.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:26.339630+0000) 2019-10-18T03:11:26.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:26.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:26.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:26.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:26.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:11:26.674+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:26.681312+0000) 2019-10-18T03:11:27.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:27.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:28.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:28.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:29.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:11:29.334+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:29.341474+0000) 2019-10-18T03:11:29.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:29.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:30.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:30.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:31.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:31.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:31.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:31.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:32.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:32.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:33.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:33.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:34.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:34.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:34.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:34.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:35.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:35.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:35.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:35.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:35.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:11:35.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:35.726938+0000) 2019-10-18T03:11:36.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:11:36.334+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:36.339821+0000) 2019-10-18T03:11:36.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:36.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:36.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:36.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:36.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:36.606+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:36.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:11:36.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:36.681442+0000) 2019-10-18T03:11:37.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:37.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:37.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:37.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:38.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:38.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:38.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:38.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:39.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:11:39.334+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:39.341589+0000) 2019-10-18T03:11:39.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:39.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:39.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:39.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:40.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:40.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:40.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:40.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:41.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:41.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:41.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:41.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:41.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:41.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:42.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:42.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:42.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:42.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:43.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:43.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:43.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:43.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:44.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:44.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:44.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:44.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:45.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:45.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:45.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:45.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:45.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:11:45.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:45.727037+0000) 2019-10-18T03:11:46.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:11:46.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:46.340009+0000) 2019-10-18T03:11:46.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:46.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:46.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:46.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:46.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:46.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:46.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:11:46.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:46.681561+0000) 2019-10-18T03:11:47.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:47.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:47.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:48.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:48.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:48.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:48.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:49.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:11:49.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:49.341699+0000) 2019-10-18T03:11:49.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:49.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:49.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:49.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:50.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:50.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:50.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:50.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:51.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:51.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:51.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:51.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:51.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:51.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:52.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:52.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:52.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:52.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:53.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:53.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:53.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:53.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:54.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:54.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:55.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:55.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:55.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:55.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:55.726 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:11:55.721+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:55.727148+0000) 2019-10-18T03:11:56.339 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:11:56.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:56.340192+0000) 2019-10-18T03:11:56.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:11:56.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:11:56.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:11:56.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:56.681679+0000) 2019-10-18T03:11:56.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:56.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:56.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:56.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:57.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:57.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:57.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:57.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:58.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:58.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:58.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:58.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:11:59.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:11:59.337+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:11:59.341798+0000) 2019-10-18T03:11:59.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:59.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:11:59.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:11:59.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:00.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:00.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:00.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:00.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:01.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:01.609+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:01.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:01.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:01.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:01.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:02.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:02.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:02.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:02.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:03.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:03.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:03.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:03.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:04.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:04.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:04.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:04.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:05.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:05.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:05.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:05.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:05.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:12:05.721+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:05.727244+0000) 2019-10-18T03:12:06.340 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:12:06.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:06.340404+0000) 2019-10-18T03:12:06.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:06.613+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:06.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:06.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:06.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:06.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:06.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:12:06.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:06.681802+0000) 2019-10-18T03:12:07.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:07.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:07.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:07.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:08.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:08.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:08.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:08.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:09.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:12:09.337+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:09.341902+0000) 2019-10-18T03:12:09.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:09.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:09.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:09.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:10.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:10.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:10.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:10.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:11.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:11.613+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:11.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:11.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:11.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:11.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:12.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:12.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:12.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:12.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:13.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:13.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:13.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:13.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:14.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:14.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:14.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:14.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:15.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:15.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:15.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:15.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:15.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:12:15.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:15.727345+0000) 2019-10-18T03:12:16.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:12:16.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:16.340588+0000) 2019-10-18T03:12:16.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:16.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:16.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:16.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:16.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:16.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:16.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:12:16.681+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:16.681933+0000) 2019-10-18T03:12:17.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:17.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:17.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:18.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:18.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:18.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:18.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:19.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:12:19.337+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:19.342021+0000) 2019-10-18T03:12:19.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:19.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:19.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:19.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:20.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:20.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:21.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:21.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:21.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:21.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:21.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:21.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:22.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:22.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:22.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:22.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:23.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:23.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:24.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:24.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:24.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:24.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:25.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:25.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:25.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:25.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:25.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:12:25.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:25.727448+0000) 2019-10-18T03:12:26.340 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:12:26.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:26.340801+0000) 2019-10-18T03:12:26.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:26.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:26.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:26.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:26.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:26.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:26.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:12:26.681+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:26.682052+0000) 2019-10-18T03:12:27.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:27.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:27.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:27.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:28.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:28.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:28.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:28.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:29.341 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:12:29.337+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:29.342133+0000) 2019-10-18T03:12:29.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:29.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:29.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:29.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:30.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:30.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:30.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:30.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:31.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:31.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:31.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:31.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:31.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:31.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:32.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:32.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:32.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:32.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:33.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:33.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:33.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:33.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:34.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:34.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:34.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:34.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:35.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:35.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:35.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:35.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:35.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:12:35.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:35.727546+0000) 2019-10-18T03:12:36.343 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:12:36.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:36.341009+0000) 2019-10-18T03:12:36.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:36.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:36.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:36.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:36.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:36.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:36.681 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:12:36.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:36.682166+0000) 2019-10-18T03:12:37.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:37.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:37.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:37.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:38.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:38.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:38.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:38.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:39.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:12:39.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:39.342251+0000) 2019-10-18T03:12:39.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:39.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:39.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:39.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:40.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:40.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:40.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:40.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:41.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:41.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:41.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:41.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:41.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:41.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:42.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:42.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:42.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:42.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:43.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:43.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:43.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:43.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:44.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:44.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:44.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:44.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:45.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:12:45.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:45.727641+0000) 2019-10-18T03:12:45.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:45.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:45.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:45.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:46.341 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:12:46.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:46.341153+0000) 2019-10-18T03:12:46.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:46.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:46.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:12:46.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:46.682280+0000) 2019-10-18T03:12:46.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:46.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:46.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:46.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:47.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:47.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:47.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:47.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:48.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:48.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:48.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:48.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:49.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:12:49.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:49.342354+0000) 2019-10-18T03:12:49.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:49.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:49.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:49.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:50.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:50.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:50.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:50.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:51.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:51.620+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:51.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:51.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:51.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:51.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:52.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:52.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:52.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:52.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:53.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:53.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:53.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:53.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:54.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:54.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:54.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:54.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:55.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:12:55.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:55.727741+0000) 2019-10-18T03:12:55.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:55.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:55.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:55.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:56.341 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:12:56.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:56.341384+0000) 2019-10-18T03:12:56.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:12:56.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:12:56.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:12:56.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:56.682405+0000) 2019-10-18T03:12:56.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:56.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:56.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:56.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:57.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:57.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:57.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:57.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:58.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:58.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:58.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:58.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:12:59.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:12:59.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:12:59.342453+0000) 2019-10-18T03:12:59.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:59.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:12:59.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:12:59.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:00.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:00.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:00.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:00.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:01.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:01.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:01.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:01.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:01.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:01.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:02.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:02.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:02.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:02.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:03.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:03.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:03.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:03.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:04.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:04.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:04.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:04.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:05.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:13:05.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:05.727838+0000) 2019-10-18T03:13:05.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:05.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:05.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:05.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:06.341 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:13:06.337+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:06.341532+0000) 2019-10-18T03:13:06.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:06.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:06.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:13:06.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:06.682526+0000) 2019-10-18T03:13:06.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:06.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:06.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:06.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:07.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:07.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:07.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:07.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:08.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:08.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:08.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:08.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:09.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:13:09.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:09.342550+0000) 2019-10-18T03:13:09.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:09.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:09.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:09.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:10.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:10.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:10.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:10.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:11.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:11.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:11.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:11.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:11.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:11.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:12.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:12.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:12.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:12.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:13.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:13.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:13.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:13.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:14.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:14.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:15.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:13:15.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:15.727936+0000) 2019-10-18T03:13:15.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:15.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:15.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:15.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:16.341 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:13:16.336+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:16.341716+0000) 2019-10-18T03:13:16.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:16.623+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:16.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:13:16.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:16.682646+0000) 2019-10-18T03:13:16.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:16.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:16.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:16.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:17.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:17.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:17.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:17.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:18.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:18.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:18.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:18.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:19.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:13:19.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:19.342684+0000) 2019-10-18T03:13:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:19.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:19.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:19.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:20.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:20.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:20.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:20.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:21.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:21.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:21.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:21.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:21.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:21.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:22.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:22.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:22.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:22.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:23.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:23.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:23.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:23.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:24.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:24.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:24.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:24.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:25.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:13:25.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:25.728035+0000) 2019-10-18T03:13:25.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:25.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:25.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:25.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:26.341 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:13:26.336+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:26.341921+0000) 2019-10-18T03:13:26.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:26.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:26.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:13:26.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:26.682770+0000) 2019-10-18T03:13:26.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:26.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:26.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:26.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:27.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:27.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:27.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:27.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:28.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:28.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:28.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:28.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:29.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:13:29.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:29.342795+0000) 2019-10-18T03:13:29.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:29.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:29.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:29.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:30.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:30.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:30.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:30.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:31.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:31.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:31.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:31.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:31.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:31.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:32.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:32.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:32.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:32.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:33.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:33.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:33.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:33.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:34.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:34.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:34.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:34.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:35.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:35.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:35.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:35.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:35.727 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:13:35.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:35.728133+0000) 2019-10-18T03:13:36.341 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:13:36.336+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:36.342105+0000) 2019-10-18T03:13:36.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:36.627+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:36.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:13:36.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:36.682902+0000) 2019-10-18T03:13:36.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:36.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:36.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:36.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:37.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:37.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:37.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:37.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:38.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:38.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:38.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:38.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:39.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:13:39.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:39.342907+0000) 2019-10-18T03:13:39.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:39.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:39.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:39.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:40.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:40.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:40.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:40.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:41.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:41.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:41.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:41.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:41.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:41.626+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:42.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:42.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:42.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:42.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:43.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:43.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:43.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:43.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:44.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:44.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:44.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:44.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:45.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:45.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:45.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:45.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:45.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:13:45.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:45.728231+0000) 2019-10-18T03:13:46.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:13:46.340+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:46.342283+0000) 2019-10-18T03:13:46.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:46.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:46.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:46.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:46.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:46.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:46.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:13:46.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:46.683025+0000) 2019-10-18T03:13:47.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:47.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:47.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:47.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:48.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:48.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:48.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:48.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:49.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:13:49.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:49.343015+0000) 2019-10-18T03:13:49.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:49.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:49.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:49.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:50.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:50.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:50.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:50.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:51.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:51.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:51.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:51.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:51.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:51.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:52.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:52.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:53.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:53.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:53.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:53.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:54.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:54.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:55.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:55.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:55.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:55.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:55.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:13:55.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:55.728330+0000) 2019-10-18T03:13:56.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:13:56.340+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:56.342489+0000) 2019-10-18T03:13:56.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:13:56.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:13:56.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:13:56.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:56.683158+0000) 2019-10-18T03:13:56.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:56.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:56.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:56.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:57.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:57.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:57.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:57.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:58.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:58.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:58.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:58.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:13:59.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:13:59.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:13:59.343127+0000) 2019-10-18T03:13:59.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:59.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:13:59.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:13:59.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:00.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:00.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:00.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:00.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:01.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:01.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:01.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:01.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:01.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:01.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:02.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:02.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:02.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:02.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:03.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:03.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:03.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:03.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:04.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:04.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:04.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:04.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:05.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:14:05.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:05.728426+0000) 2019-10-18T03:14:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:05.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:05.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:06.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:14:06.340+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:06.342670+0000) 2019-10-18T03:14:06.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:06.630+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:06.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:14:06.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:06.683278+0000) 2019-10-18T03:14:06.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:06.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:06.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:06.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:07.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:07.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:07.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:07.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:08.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:08.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:08.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:08.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:09.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:14:09.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:09.343233+0000) 2019-10-18T03:14:09.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:09.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:09.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:09.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:10.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:10.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:11.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:11.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:11.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:11.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:11.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:11.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:12.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:12.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:12.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:12.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:13.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:13.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:13.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:13.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:14.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:14.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:14.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:14.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:15.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:14:15.722+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:15.728520+0000) 2019-10-18T03:14:15.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:15.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:15.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:15.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:16.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:14:16.340+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:16.342849+0000) 2019-10-18T03:14:16.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:16.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:16.682 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:14:16.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:16.683409+0000) 2019-10-18T03:14:16.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:16.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:16.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:16.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:17.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:17.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:17.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:17.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:18.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:18.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:18.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:18.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:19.342 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:14:19.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:19.343349+0000) 2019-10-18T03:14:19.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:19.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:19.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:19.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:20.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:20.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:20.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:20.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:21.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:21.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:21.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:21.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:21.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:21.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:22.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:22.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:23.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:23.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:23.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:23.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:24.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:24.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:24.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:24.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:25.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:14:25.721+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:25.728615+0000) 2019-10-18T03:14:25.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:25.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:25.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:25.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:26.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:14:26.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:26.343030+0000) 2019-10-18T03:14:26.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:26.633+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:26.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:14:26.677+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:26.683527+0000) 2019-10-18T03:14:26.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:26.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:26.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:26.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:27.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:27.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:27.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:27.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:28.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:28.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:28.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:28.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:29.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:14:29.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:29.343444+0000) 2019-10-18T03:14:29.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:29.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:29.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:29.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:30.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:30.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:31.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:31.637+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:31.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:31.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:31.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:31.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:32.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:32.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:32.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:32.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:33.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:33.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:33.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:33.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:34.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:34.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:34.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:34.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:35.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:14:35.721+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:35.728707+0000) 2019-10-18T03:14:35.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:35.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:35.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:35.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:36.342 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:14:36.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:36.343217+0000) 2019-10-18T03:14:36.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:36.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:36.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:14:36.681+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:36.683645+0000) 2019-10-18T03:14:36.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:36.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:36.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:36.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:37.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:37.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:37.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:37.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:38.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:38.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:38.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:38.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:39.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:14:39.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:39.343563+0000) 2019-10-18T03:14:39.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:39.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:39.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:39.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:40.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:40.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:40.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:40.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:41.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:41.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:41.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:41.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:41.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:41.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:42.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:42.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:42.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:42.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:43.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:43.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:43.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:43.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:44.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:44.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:44.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:44.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 336 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:45.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:14:45.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:45.728804+0000) 2019-10-18T03:14:45.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:45.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:45.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:45.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:46.343 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:14:46.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:46.343427+0000) 2019-10-18T03:14:46.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:46.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:46.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:14:46.681+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:46.683760+0000) 2019-10-18T03:14:46.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:46.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:46.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:46.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:47.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:47.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:47.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:47.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:48.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:48.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:48.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:48.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:49.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:14:49.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:49.343678+0000) 2019-10-18T03:14:49.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:49.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:49.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:49.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:50.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:50.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:50.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:50.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:51.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:51.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:51.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:51.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:51.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:51.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:52.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:52.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:52.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:52.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:53.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:53.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:53.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:53.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:54.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:54.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:54.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:54.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:55.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:14:55.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:55.728906+0000) 2019-10-18T03:14:55.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:55.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:55.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:55.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:56.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:14:56.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:56.343610+0000) 2019-10-18T03:14:56.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:14:56.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:14:56.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:14:56.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:56.683881+0000) 2019-10-18T03:14:56.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:56.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:56.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:56.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:57.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:57.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:57.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:57.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:58.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:58.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:58.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:58.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:14:59.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:14:59.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:14:59.343792+0000) 2019-10-18T03:14:59.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:59.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:14:59.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:14:59.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:00.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:00.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:00.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:00.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:01.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:01.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:01.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:01.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:01.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:01.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:02.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:02.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:03.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:03.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:03.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:03.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:04.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:04.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:04.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:04.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:05.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:15:05.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:05.729005+0000) 2019-10-18T03:15:05.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:05.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:05.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:05.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:06.343 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:15:06.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:06.343798+0000) 2019-10-18T03:15:06.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:06.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:06.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:15:06.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:06.683991+0000) 2019-10-18T03:15:06.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:06.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:06.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:06.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:07.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:07.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:07.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:07.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:08.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:08.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:08.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:08.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:09.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:15:09.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:09.343904+0000) 2019-10-18T03:15:09.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:09.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:09.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:09.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:10.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:10.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:11.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:11.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:11.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:11.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:11.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:11.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:12.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:12.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:12.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:12.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:13.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:13.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:13.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:13.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:14.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:14.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:14.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:14.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:15.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:15:15.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:15.729099+0000) 2019-10-18T03:15:15.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:15.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:15.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:15.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:16.351 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:15:16.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:16.343980+0000) 2019-10-18T03:15:16.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:16.644+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:16.683 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:15:16.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:16.684107+0000) 2019-10-18T03:15:16.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:16.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:16.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:16.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:17.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:17.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:17.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:17.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:18.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:18.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:18.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:18.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:19.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:15:19.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:19.344012+0000) 2019-10-18T03:15:19.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:19.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:19.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:19.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:20.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:20.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:20.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:20.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:21.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:21.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:21.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:21.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:21.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:21.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:22.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:22.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:22.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:22.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:23.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:23.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:23.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:23.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:24.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:24.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:24.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:24.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:25.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:15:25.724+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:25.729195+0000) 2019-10-18T03:15:25.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:25.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:25.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:25.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:26.343 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:15:26.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:26.344161+0000) 2019-10-18T03:15:26.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:26.648+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:26.684 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:15:26.680+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:26.684231+0000) 2019-10-18T03:15:26.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:26.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:26.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:26.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:27.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:27.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:27.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:27.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:28.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:28.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:28.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:28.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:29.343 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:15:29.339+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:29.344120+0000) 2019-10-18T03:15:29.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:29.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:29.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:29.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:30.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:30.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:30.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:30.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:31.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:31.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:31.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:31.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:31.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:31.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:32.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:32.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:32.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:32.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:33.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:33.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:33.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:33.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:34.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:34.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:34.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:34.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:35.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:15:35.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:35.729293+0000) 2019-10-18T03:15:35.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:35.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:35.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:35.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:36.344 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:15:36.339+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:36.344342+0000) 2019-10-18T03:15:36.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:36.647+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:36.684 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:15:36.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:36.684355+0000) 2019-10-18T03:15:36.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:36.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:36.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:36.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:37.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:37.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:37.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:37.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:38.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:38.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:38.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:38.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:39.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:15:39.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:39.344255+0000) 2019-10-18T03:15:39.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:39.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:39.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:39.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:40.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:40.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:40.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:40.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:41.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:41.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:41.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:41.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:41.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:41.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:42.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:42.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:42.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:42.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:43.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:43.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:43.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:43.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:44.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:44.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:44.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:44.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:45.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:15:45.723+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:45.729388+0000) 2019-10-18T03:15:45.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:45.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:45.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:45.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:46.344 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:15:46.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:46.344533+0000) 2019-10-18T03:15:46.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:46.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:46.684 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:15:46.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:46.684470+0000) 2019-10-18T03:15:46.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:46.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:46.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:46.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:47.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:47.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:47.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:47.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:48.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:48.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:48.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:48.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:49.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:15:49.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:49.344371+0000) 2019-10-18T03:15:49.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:49.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:49.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:49.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:50.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:50.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:50.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:50.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:51.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:51.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:51.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:51.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:51.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:51.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:52.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:52.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:52.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:52.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:53.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:53.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:53.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:53.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:54.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:54.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:54.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:54.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:55.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:55.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:55.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:55.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:55.728 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:15:55.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:55.729466+0000) 2019-10-18T03:15:56.348 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:15:56.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:56.344722+0000) 2019-10-18T03:15:56.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:15:56.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:15:56.684 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:15:56.679+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:56.684633+0000) 2019-10-18T03:15:56.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:56.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:56.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:56.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:57.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:57.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:57.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:57.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:58.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:58.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:58.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:15:59.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:15:59.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:15:59.344480+0000) 2019-10-18T03:15:59.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:59.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:15:59.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:15:59.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:00.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:00.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:00.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:00.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:01.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:01.651+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:01.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:01.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:01.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:01.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:02.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:02.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:02.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:02.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:03.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:03.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:03.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:03.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:04.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:04.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:04.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:04.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:05.741 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:16:05.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:05.729562+0000) 2019-10-18T03:16:05.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:05.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:05.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:05.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:06.344 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:16:06.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:06.344901+0000) 2019-10-18T03:16:06.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:06.650+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:06.685 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:16:06.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:06.684788+0000) 2019-10-18T03:16:06.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:06.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:06.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:06.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:07.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:07.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:07.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:07.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:08.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:08.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:08.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:08.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:09.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:16:09.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:09.344594+0000) 2019-10-18T03:16:09.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:09.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:09.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:09.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:10.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:10.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:10.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:10.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:11.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:11.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:11.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:11.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:11.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:11.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:12.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:12.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:12.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:12.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:13.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:13.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:13.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:13.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:14.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:14.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:14.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:14.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:15.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:16:15.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:15.729659+0000) 2019-10-18T03:16:15.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:15.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:15.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:15.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:16.344 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:16:16.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:16.345076+0000) 2019-10-18T03:16:16.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:16.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:16.684 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:16:16.678+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:16.684921+0000) 2019-10-18T03:16:16.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:16.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:16.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:16.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:17.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:17.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:17.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:17.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:18.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:18.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:18.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:18.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:19.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:16:19.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:19.344712+0000) 2019-10-18T03:16:19.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:19.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:19.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:19.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:20.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:20.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:20.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:20.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:21.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:21.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:21.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:21.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:21.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:21.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:22.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:22.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:22.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:22.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:23.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:23.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:23.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:24.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:24.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:24.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:24.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:25.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:16:25.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:25.729760+0000) 2019-10-18T03:16:25.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:25.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:25.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:25.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:26.344 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:16:26.338+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:26.345255+0000) 2019-10-18T03:16:26.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:26.654+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:26.689 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:16:26.686+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:26.689558+0000) 2019-10-18T03:16:26.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:26.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:26.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:26.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:27.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:27.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:27.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:27.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:28.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:28.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:29.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:28.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:29.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:16:29.338+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:29.344816+0000) 2019-10-18T03:16:30.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:30.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:30.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:30.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:31.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:31.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:31.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:31.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:31.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:31.658+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:32.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:32.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:32.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:32.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:33.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:33.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:33.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:33.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:34.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:34.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:34.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:34.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:35.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:35.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:35.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:35.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:35.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:16:35.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:35.729876+0000) 2019-10-18T03:16:36.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:36.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:36.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:36.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:36.345 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:16:36.342+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:36.345441+0000) 2019-10-18T03:16:36.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:36.658+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:36.690 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:16:36.686+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:36.689691+0000) 2019-10-18T03:16:37.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:37.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:37.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:37.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:38.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:38.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:38.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:38.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:39.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:39.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:39.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:39.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:39.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:16:39.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:39.344928+0000) 2019-10-18T03:16:40.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:40.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:40.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:40.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:40.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:40.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:40.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:40.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:41.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:41.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:41.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:41.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:41.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:41.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:42.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:42.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:42.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:42.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:44.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:44.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:44.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:44.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:44.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:44.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:44.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:44.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:45.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:16:45.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:45.729972+0000) 2019-10-18T03:16:45.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:45.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:45.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:45.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:46.345 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:16:46.342+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:46.345625+0000) 2019-10-18T03:16:46.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:46.657+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:46.693 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:16:46.689+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:46.693515+0000) 2019-10-18T03:16:46.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:46.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:46.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:46.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:47.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:47.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:47.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:47.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:48.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:48.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:48.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:48.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:49.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:16:49.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:49.345046+0000) 2019-10-18T03:16:49.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:49.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:49.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:49.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:50.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:50.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:50.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:50.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:51.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:51.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:51.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:51.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:51.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:51.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:52.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:52.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:52.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:52.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:53.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:53.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:53.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:53.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:54.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:54.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:54.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:54.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:55.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:16:55.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:55.730072+0000) 2019-10-18T03:16:55.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:55.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:55.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:55.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:56.345 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:16:56.341+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:56.345810+0000) 2019-10-18T03:16:56.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:16:56.661+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:16:56.693 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:16:56.689+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:56.693650+0000) 2019-10-18T03:16:56.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:56.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:56.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:56.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:57.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:57.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:57.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:57.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:58.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:58.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:16:59.344 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:16:59.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:16:59.345153+0000) 2019-10-18T03:16:59.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:59.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:16:59.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:16:59.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:00.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:00.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:00.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:00.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:01.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:01.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:01.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:01.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:01.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:01.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:02.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:02.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:02.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:02.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:03.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:03.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:03.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:03.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:04.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:04.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:04.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:04.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:05.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:17:05.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:05.730168+0000) 2019-10-18T03:17:05.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:05.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:05.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:05.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:06.345 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:17:06.341+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:06.345997+0000) 2019-10-18T03:17:06.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:06.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:06.694 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:17:06.693+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:06.694803+0000) 2019-10-18T03:17:06.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:06.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:06.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:06.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:07.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:07.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:07.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:07.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:08.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:08.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:08.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:08.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:09.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:17:09.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:09.345259+0000) 2019-10-18T03:17:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:09.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:09.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:10.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:10.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:10.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:10.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:11.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:11.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:11.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:11.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:11.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:11.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:12.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:12.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:12.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:12.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:13.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:13.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:13.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:13.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:14.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:14.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:14.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:14.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:15.729 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:17:15.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:15.730264+0000) 2019-10-18T03:17:15.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:15.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:15.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:15.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:16.345 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:17:16.345+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:16.346205+0000) 2019-10-18T03:17:16.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:16.664+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:16.694 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:17:16.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:16.694939+0000) 2019-10-18T03:17:16.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:16.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:16.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:16.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:17.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:17.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:17.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:17.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:18.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:18.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:18.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:18.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:19.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:17:19.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:19.345374+0000) 2019-10-18T03:17:19.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:19.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:19.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:19.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:20.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:20.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:20.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:20.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:21.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:21.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:21.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:21.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:22.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:22.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:22.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:22.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:23.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:23.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:23.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:23.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:24.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:24.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:24.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:24.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:25.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:17:25.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:25.730363+0000) 2019-10-18T03:17:25.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:25.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:25.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:25.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:26.346 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:17:26.345+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:26.346403+0000) 2019-10-18T03:17:26.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:26.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:26.694 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:17:26.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:26.695079+0000) 2019-10-18T03:17:26.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:26.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:26.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:26.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:27.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:27.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:27.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:27.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:28.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:28.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:28.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:28.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:29.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:17:29.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:29.345488+0000) 2019-10-18T03:17:29.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:29.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:29.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:29.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:30.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:30.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:30.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:30.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:31.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:31.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:31.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:31.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:31.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:31.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:32.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:32.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:32.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:32.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:33.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:33.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:33.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:33.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:34.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:34.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:34.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:34.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:35.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:17:35.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:35.730463+0000) 2019-10-18T03:17:35.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:35.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:35.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:35.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:36.346 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:17:36.345+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:36.346610+0000) 2019-10-18T03:17:36.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:36.668+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:36.694 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:17:36.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:36.695209+0000) 2019-10-18T03:17:36.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:36.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:36.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:36.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:37.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:37.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:37.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:37.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:38.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:38.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:38.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:38.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:39.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:17:39.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:39.345603+0000) 2019-10-18T03:17:39.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:39.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:39.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:39.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:40.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:40.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:40.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:41.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:41.672+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:41.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:41.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:41.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:41.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:42.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:42.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:42.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:42.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:43.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:43.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:43.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:43.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:44.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:44.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:44.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:44.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:45.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:17:45.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:45.730568+0000) 2019-10-18T03:17:45.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:45.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:45.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:45.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:46.346 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:17:46.345+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:46.346790+0000) 2019-10-18T03:17:46.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:46.672+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:46.694 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:17:46.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:46.695343+0000) 2019-10-18T03:17:46.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:46.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:46.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:46.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:47.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:47.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:47.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:47.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:48.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:48.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:48.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:48.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:49.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:17:49.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:49.345717+0000) 2019-10-18T03:17:49.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:49.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:49.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:49.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:50.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:50.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:50.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:50.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:51.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:51.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:51.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:51.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:51.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:51.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:52.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:52.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:52.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:52.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:53.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:53.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:53.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:53.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:54.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:54.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:54.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:54.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:55.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:17:55.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:55.730670+0000) 2019-10-18T03:17:55.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:55.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:55.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:55.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:56.348 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:17:56.345+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:56.346937+0000) 2019-10-18T03:17:56.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:17:56.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:17:56.695 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:17:56.691+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:56.695486+0000) 2019-10-18T03:17:56.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:56.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:56.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:56.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:57.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:57.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:57.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:57.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:58.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:58.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:58.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:58.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:17:59.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:17:59.341+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:17:59.345829+0000) 2019-10-18T03:17:59.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:59.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:17:59.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:17:59.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:00.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:00.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:00.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:00.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:01.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:01.671+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:01.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:01.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:01.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:01.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:02.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:02.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:02.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:02.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:03.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:03.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:03.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:03.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:04.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:04.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:04.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:04.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:05.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:18:05.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:05.730767+0000) 2019-10-18T03:18:05.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:05.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:05.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:05.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:06.346 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:18:06.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:06.347111+0000) 2019-10-18T03:18:06.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:06.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:06.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:18:06.691+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:06.695633+0000) 2019-10-18T03:18:06.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:06.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:06.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:06.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:07.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:07.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:07.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:07.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:08.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:08.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:08.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:08.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:09.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:18:09.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:09.345937+0000) 2019-10-18T03:18:09.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:09.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:09.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:09.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:10.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:10.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:10.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:10.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:11.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:11.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:11.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:11.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:11.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:12.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:12.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:12.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:12.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:13.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:13.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:13.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:13.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:14.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:14.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:15.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:18:15.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:15.730860+0000) 2019-10-18T03:18:15.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:15.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:15.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:15.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:16.346 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:18:16.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:16.347298+0000) 2019-10-18T03:18:16.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:16.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:16.695 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:18:16.691+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:16.695768+0000) 2019-10-18T03:18:17.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:17.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:17.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:17.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:18.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:18.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:18.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:18.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:19.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:19.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:19.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:19.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:19.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:18:19.340+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:19.346047+0000) 2019-10-18T03:18:20.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:20.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:20.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:20.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:21.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:21.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:21.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:21.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:21.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:21.675+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:22.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:22.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:22.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:22.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:23.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:23.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:23.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:23.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:24.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:24.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:25.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:25.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:25.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:25.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:25.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:18:25.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:25.730959+0000) 2019-10-18T03:18:26.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:26.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:26.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:26.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:26.347 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:18:26.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:26.347487+0000) 2019-10-18T03:18:26.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:26.674+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:26.695 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:18:26.690+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:26.695921+0000) 2019-10-18T03:18:27.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:27.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:27.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:27.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:27.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:27.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:27.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:27.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:28.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:28.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:28.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:28.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:29.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:18:29.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:29.346157+0000) 2019-10-18T03:18:30.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:30.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:30.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:30.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:30.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:30.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:30.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:30.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:31.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:31.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:31.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:31.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:31.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:31.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:32.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:32.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:32.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:32.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:33.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:33.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:33.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:33.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:34.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:34.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:34.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:34.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:35.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:18:35.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:35.731054+0000) 2019-10-18T03:18:35.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:35.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:35.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:35.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:36.347 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:18:36.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:36.347668+0000) 2019-10-18T03:18:36.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:36.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:36.695 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:18:36.690+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:36.696050+0000) 2019-10-18T03:18:36.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:36.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:36.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:36.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:38.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:37.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:38.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:37.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:38.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:38.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:38.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:38.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:39.345 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:18:39.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:39.346275+0000) 2019-10-18T03:18:39.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:39.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:39.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:39.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:40.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:40.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:40.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:40.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:41.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:41.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:41.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:41.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:41.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:41.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:42.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:42.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:42.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:42.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:43.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:43.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:43.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:43.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:44.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:44.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:44.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:44.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:45.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:18:45.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:45.731151+0000) 2019-10-18T03:18:45.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:45.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:45.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:45.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:46.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:18:46.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:46.347853+0000) 2019-10-18T03:18:46.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:46.678+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:46.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:18:46.690+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:46.696192+0000) 2019-10-18T03:18:46.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:46.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:46.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:46.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:47.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:47.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:47.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:47.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:48.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:48.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:48.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:48.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:49.346 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:18:49.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:49.346378+0000) 2019-10-18T03:18:49.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:49.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:49.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:49.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:51.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:50.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:51.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:50.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:51.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:51.682+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:51.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:51.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:51.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:51.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:52.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:52.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:52.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:52.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:53.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:53.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:53.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:53.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:54.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:54.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:54.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:54.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:55.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:18:55.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:55.731250+0000) 2019-10-18T03:18:55.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:55.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:55.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:55.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:56.347 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:18:56.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:56.348035+0000) 2019-10-18T03:18:56.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:18:56.682+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:18:56.695 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:18:56.690+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:56.696324+0000) 2019-10-18T03:18:56.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:56.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:56.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:56.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:57.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:57.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:57.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:57.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:58.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:58.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:58.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:58.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:18:59.346 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:18:59.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:18:59.346479+0000) 2019-10-18T03:18:59.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:59.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:18:59.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:18:59.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:00.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:00.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:01.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:01.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:01.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:01.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:01.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:01.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:02.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:02.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:02.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:02.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:03.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:03.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:03.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:03.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:04.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:04.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:04.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:04.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:05.730 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:19:05.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:05.731342+0000) 2019-10-18T03:19:05.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:05.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:05.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:05.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:06.347 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:19:06.344+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:06.348223+0000) 2019-10-18T03:19:06.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:06.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:06.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:19:06.689+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:06.696495+0000) 2019-10-18T03:19:06.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:06.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:06.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:06.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:07.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:07.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:07.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:07.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:08.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:08.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:08.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:08.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:09.346 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:19:09.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:09.346577+0000) 2019-10-18T03:19:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:09.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:09.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:10.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:10.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:10.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:10.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:11.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:11.681+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:11.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:11.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:11.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:11.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:12.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:12.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:12.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:12.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:13.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:13.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:14.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:14.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:14.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:14.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:15.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:19:15.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:15.731438+0000) 2019-10-18T03:19:15.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:15.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:15.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:15.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:16.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:19:16.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:16.348411+0000) 2019-10-18T03:19:16.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:16.685+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:16.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:19:16.689+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:16.696633+0000) 2019-10-18T03:19:16.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:16.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:16.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:16.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:17.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:17.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:17.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:17.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:18.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:18.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:18.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:18.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:19.346 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:19:19.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:19.346689+0000) 2019-10-18T03:19:19.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:19.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:19.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:19.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:20.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:20.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:20.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:20.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:21.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:21.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:21.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:21.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:21.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:21.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:22.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:22.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:22.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:22.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:23.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:23.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:23.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:23.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:24.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:24.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:24.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:24.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:25.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:19:25.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:25.731534+0000) 2019-10-18T03:19:25.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:25.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:25.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:25.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:26.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:19:26.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:26.348594+0000) 2019-10-18T03:19:26.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:26.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:26.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:19:26.693+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:26.696779+0000) 2019-10-18T03:19:26.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:26.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:26.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:26.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:27.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:27.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:27.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:27.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:28.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:28.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:28.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:28.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:29.346 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:19:29.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:29.346793+0000) 2019-10-18T03:19:29.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:29.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:29.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:29.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:30.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:30.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:30.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:30.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:31.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:31.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:31.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:31.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:31.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:31.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:32.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:32.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:32.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:32.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:33.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:33.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:33.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:33.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:34.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:34.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:34.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:34.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:35.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:19:35.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:35.731629+0000) 2019-10-18T03:19:35.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:35.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:35.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:35.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:36.348 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:19:36.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:36.348782+0000) 2019-10-18T03:19:36.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:36.688+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:36.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:19:36.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:36.696904+0000) 2019-10-18T03:19:36.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:36.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:36.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:36.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:37.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:37.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:37.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:37.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:38.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:38.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:38.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:38.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:39.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:19:39.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:39.346907+0000) 2019-10-18T03:19:39.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:39.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:39.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:39.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:40.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:40.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:40.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:40.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:41.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:41.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:41.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:41.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:41.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:41.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:42.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:42.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:42.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:42.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:43.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:43.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:43.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:43.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:44.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:44.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:44.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:44.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:45.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:19:45.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:45.731725+0000) 2019-10-18T03:19:45.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:45.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:45.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:45.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:46.348 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:19:46.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:46.348964+0000) 2019-10-18T03:19:46.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:46.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:46.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:19:46.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:46.697179+0000) 2019-10-18T03:19:46.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:46.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:46.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:46.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:47.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:47.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:47.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:47.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:48.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:48.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:48.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:48.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:49.346 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:19:49.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:49.347015+0000) 2019-10-18T03:19:50.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:50.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:50.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:50.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:51.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:51.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:51.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:51.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:51.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:51.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:52.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:51.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:52.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:51.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:52.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:52.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:52.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:52.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:53.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:53.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:53.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:53.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:54.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:54.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:54.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:54.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:55.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:19:55.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:55.731825+0000) 2019-10-18T03:19:55.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:55.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:55.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:55.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:56.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:19:56.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:56.349177+0000) 2019-10-18T03:19:56.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:19:56.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:19:56.696 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:19:56.692+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:56.697323+0000) 2019-10-18T03:19:56.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:56.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:56.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:56.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:57.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:57.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:57.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:57.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:58.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:58.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:58.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:58.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:19:59.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:19:59.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:19:59.347134+0000) 2019-10-18T03:19:59.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:59.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:19:59.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:19:59.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:00.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:00.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:00.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:00.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:01.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:01.692+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:01.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:01.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:01.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:01.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:02.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:02.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:02.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:02.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:03.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:03.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:03.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:03.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:04.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:04.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:04.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:04.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:05.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:20:05.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:05.731925+0000) 2019-10-18T03:20:06.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:06.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:06.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:06.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:06.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:20:06.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:06.349358+0000) 2019-10-18T03:20:06.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:20:06.696+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:06.697456+0000) 2019-10-18T03:20:06.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:06.696+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:07.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:06.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:07.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:06.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:08.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:08.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:08.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:08.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:09.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:09.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:09.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:09.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:09.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:20:09.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:09.347249+0000) 2019-10-18T03:20:10.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:10.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:10.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:10.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:11.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:11.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:11.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:11.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:11.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:11.696+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:12.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:12.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:12.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:12.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:13.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:13.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:13.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:13.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:14.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:14.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:14.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:14.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:15.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:15.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:15.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:15.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:15.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:20:15.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:15.732020+0000) 2019-10-18T03:20:15.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:15.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:15.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:15.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:16.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:20:16.343+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:16.349543+0000) 2019-10-18T03:20:16.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:20:16.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:16.697598+0000) 2019-10-18T03:20:16.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:16.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:17.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:16.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:17.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:16.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:17.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:17.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:17.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:17.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:18.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:18.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:18.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:18.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:19.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:20:19.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:19.347378+0000) 2019-10-18T03:20:19.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:19.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:19.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:19.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:21.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:21.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:21.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:21.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:21.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:21.695+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:22.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:22.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:22.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:22.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:22.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:22.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:22.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:22.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:23.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:23.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:24.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:24.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:24.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:24.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:25.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:20:25.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:25.732124+0000) 2019-10-18T03:20:25.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:25.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:25.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:25.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:26.350 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:20:26.342+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:26.349729+0000) 2019-10-18T03:20:26.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:20:26.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:26.697721+0000) 2019-10-18T03:20:26.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:26.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:26.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:26.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:26.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:26.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:27.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:27.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:27.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:28.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:28.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:28.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:28.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:29.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:20:29.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:29.347471+0000) 2019-10-18T03:20:29.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:29.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:29.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:29.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:30.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:30.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:30.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:30.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:31.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:31.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:31.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:31.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:31.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:31.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:32.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:32.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:32.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:32.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:33.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:33.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:33.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:33.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:34.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:34.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:34.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:34.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:35.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:20:35.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:35.732228+0000) 2019-10-18T03:20:35.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:35.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:35.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:35.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:36.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:20:36.342+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:36.349912+0000) 2019-10-18T03:20:36.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:20:36.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:36.697833+0000) 2019-10-18T03:20:36.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:36.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:36.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:36.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:36.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:36.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:37.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:37.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:37.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:37.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:38.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:38.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:38.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:38.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:39.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:20:39.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:39.347567+0000) 2019-10-18T03:20:39.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:39.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:39.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:39.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:40.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:40.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:40.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:40.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:41.703 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:41.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:41.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:41.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:41.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:41.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:42.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:42.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:42.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:42.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:43.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:43.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:43.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:43.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:44.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:44.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:44.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:44.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:45.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:20:45.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:45.732324+0000) 2019-10-18T03:20:45.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:45.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:45.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:45.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:46.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:20:46.342+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:46.350091+0000) 2019-10-18T03:20:46.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:20:46.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:46.697952+0000) 2019-10-18T03:20:46.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:46.699+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:46.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:46.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:46.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:46.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:47.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:47.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:47.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:47.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:48.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:48.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:48.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:48.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:49.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:20:49.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:49.347683+0000) 2019-10-18T03:20:49.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:49.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:49.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:49.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:50.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:50.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:50.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:50.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:51.705 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:51.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:51.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:51.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:51.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:51.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:52.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:52.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:52.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:52.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:53.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:53.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:53.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:53.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:54.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:54.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:54.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:54.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:55.731 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:20:55.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:55.732423+0000) 2019-10-18T03:20:55.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:55.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:55.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:55.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:56.349 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:20:56.346+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:56.350270+0000) 2019-10-18T03:20:56.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:20:56.694+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:56.698083+0000) 2019-10-18T03:20:56.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:20:56.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:20:56.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:56.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:56.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:56.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:57.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:57.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:57.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:57.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:59.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:59.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:20:59.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:59.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:20:59.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:20:59.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:20:59.347803+0000) 2019-10-18T03:20:59.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:59.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:00.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:20:59.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:01.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:01.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:01.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:01.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:01.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:01.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:02.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:02.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:02.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:02.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:03.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:03.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:03.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:03.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:04.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:03.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:04.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:03.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:04.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:04.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:04.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:04.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:05.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:21:05.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:05.732528+0000) 2019-10-18T03:21:05.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:05.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:05.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:05.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:06.350 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:21:06.346+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:06.350474+0000) 2019-10-18T03:21:06.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:21:06.694+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:06.698212+0000) 2019-10-18T03:21:06.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:06.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:06.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:06.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:06.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:06.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:07.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:07.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:07.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:07.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:08.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:08.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:08.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:08.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:09.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:21:09.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:09.347916+0000) 2019-10-18T03:21:09.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:09.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:09.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:09.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:10.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:10.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:10.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:10.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:11.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:11.702+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:11.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:11.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:12.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:12.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:12.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:12.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:13.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:13.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:13.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:13.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:14.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:14.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:15.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:21:15.726+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:15.732640+0000) 2019-10-18T03:21:16.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:16.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:16.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:16.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:16.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:21:16.346+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:16.350663+0000) 2019-10-18T03:21:16.697 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:21:16.694+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:16.698333+0000) 2019-10-18T03:21:16.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:16.706+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:17.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:17.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:17.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:17.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:18.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:18.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:18.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:18.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:18.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:18.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:18.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:18.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:19.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:21:19.342+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:19.348034+0000) 2019-10-18T03:21:19.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:19.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:19.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:19.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:21.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:21.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:21.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:21.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:21.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:21.706+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:21.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:21.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:21.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:21.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:22.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:22.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:22.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:22.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:23.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:23.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:23.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:23.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:24.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:24.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:24.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:24.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:25.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:21:25.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:25.732736+0000) 2019-10-18T03:21:26.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:26.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:26.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:26.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:26.350 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:21:26.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:26.350845+0000) 2019-10-18T03:21:26.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:21:26.693+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:26.698452+0000) 2019-10-18T03:21:26.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:26.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:27.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:27.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:27.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:27.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:28.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:28.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:28.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:28.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:29.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:29.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:29.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:29.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:29.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:21:29.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:29.348150+0000) 2019-10-18T03:21:30.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:30.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:30.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:30.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:31.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:31.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:31.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:31.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:31.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:31.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:32.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:32.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:32.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:32.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:33.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:33.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:33.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:33.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:34.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:34.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:34.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:34.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:35.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:35.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:35.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:35.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:35.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:21:35.725+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:35.732833+0000) 2019-10-18T03:21:36.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:36.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:36.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:36.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:36.350 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:21:36.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:36.351022+0000) 2019-10-18T03:21:36.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:21:36.693+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:36.698571+0000) 2019-10-18T03:21:36.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:36.705+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:37.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:37.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:37.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:37.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:38.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:38.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:38.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:38.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:39.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:39.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:39.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:39.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:39.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:21:39.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:39.348268+0000) 2019-10-18T03:21:40.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:40.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:40.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:40.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:41.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:41.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:41.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:41.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:41.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:41.709+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:42.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:42.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:42.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:42.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:43.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:43.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:44.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:44.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:11:14.539062+0000 (oldest deadline 2019-10-18T03:11:34.539062+0000) 2019-10-18T03:21:44.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:44.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:45.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:45.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:45.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:21:45.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:45.732928+0000) 2019-10-18T03:21:46.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:46.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:46.350 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:21:46.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:46.351207+0000) 2019-10-18T03:21:46.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:21:46.697+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:46.698691+0000) 2019-10-18T03:21:46.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:46.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:47.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:47.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:48.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:48.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:49.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:49.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:49.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:21:49.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:49.348379+0000) 2019-10-18T03:21:50.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:50.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:51.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:51.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:51.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:51.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:52.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:52.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:53.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:53.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:54.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:54.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:55.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:55.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:55.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:21:55.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:55.733025+0000) 2019-10-18T03:21:56.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:56.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:56.350 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:21:56.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:56.351357+0000) 2019-10-18T03:21:56.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:21:56.697+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:56.698807+0000) 2019-10-18T03:21:56.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:21:56.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:21:57.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:57.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:58.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:58.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:59.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:21:59.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:21:59.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:21:59.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:21:59.348497+0000) 2019-10-18T03:22:00.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:00.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:01.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:01.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:01.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:01.712+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:02.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:02.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:03.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:03.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:04.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:04.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:05.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:05.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:05.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:22:05.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:05.733125+0000) 2019-10-18T03:22:06.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:06.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:06.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:22:06.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:06.351536+0000) 2019-10-18T03:22:06.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:22:06.696+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:06.698925+0000) 2019-10-18T03:22:06.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:06.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:07.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:07.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:07.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:07.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:08.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:08.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:08.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:08.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:09.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:09.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:09.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:09.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:09.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:22:09.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:09.348609+0000) 2019-10-18T03:22:10.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:10.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:10.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:10.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:11.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:11.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:11.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:11.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:11.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:11.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:12.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:12.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:12.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:12.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:13.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:13.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:13.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:13.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:14.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:14.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:14.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:14.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:15.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:15.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:15.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:15.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:15.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:22:15.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:15.733219+0000) 2019-10-18T03:22:16.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:16.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:16.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:16.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:16.351 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:22:16.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:16.351742+0000) 2019-10-18T03:22:16.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:22:16.696+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:16.699046+0000) 2019-10-18T03:22:16.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:16.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:17.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:17.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:17.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:17.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:18.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:18.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:18.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:18.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:19.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:19.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:19.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:19.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:19.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:22:19.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:19.348722+0000) 2019-10-18T03:22:20.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:20.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:20.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:20.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:21.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:21.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:21.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:21.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:21.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:21.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:22.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:22.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:22.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:22.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:23.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:23.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:23.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:23.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:24.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:24.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:24.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:25.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:25.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:25.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:25.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:25.732 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:22:25.728+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:25.733314+0000) 2019-10-18T03:22:26.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:26.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:26.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:26.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:26.351 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:22:26.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:26.351929+0000) 2019-10-18T03:22:26.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:22:26.696+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:26.699173+0000) 2019-10-18T03:22:26.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:26.716+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:27.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:27.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:27.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:27.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:28.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:28.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:28.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:28.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:29.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:29.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:29.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:29.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:29.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:22:29.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:29.348829+0000) 2019-10-18T03:22:30.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:30.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:30.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:30.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:31.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:31.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:31.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:31.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:31.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:31.720+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:32.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:32.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:32.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:32.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:33.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:33.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:33.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:33.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:34.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:34.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:34.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:34.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:35.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:35.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:35.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:35.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:35.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:22:35.727+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:35.733412+0000) 2019-10-18T03:22:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:36.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:36.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:36.351 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:22:36.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:36.352110+0000) 2019-10-18T03:22:36.698 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:22:36.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:36.699305+0000) 2019-10-18T03:22:36.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:36.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:37.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:37.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:37.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:37.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:38.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:38.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:38.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:38.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:39.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:39.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:39.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:39.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:39.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:22:39.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:39.348934+0000) 2019-10-18T03:22:40.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:40.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:40.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:40.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:41.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:41.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:41.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:41.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:41.724 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:41.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:42.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:42.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:42.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:42.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:43.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:43.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:43.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:43.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:44.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:44.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:44.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:44.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:45.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:45.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:45.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:45.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:45.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:22:45.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:45.733505+0000) 2019-10-18T03:22:46.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:46.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:46.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:46.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:46.351 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:22:46.349+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:46.352292+0000) 2019-10-18T03:22:46.707 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:22:46.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:46.699418+0000) 2019-10-18T03:22:46.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:46.719+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:47.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:47.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:47.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:47.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:48.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:48.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:48.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:48.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:49.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:49.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:49.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:49.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:49.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:22:49.345+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:49.349044+0000) 2019-10-18T03:22:50.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:50.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:50.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:50.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:51.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:51.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:51.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:51.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:51.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:51.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:52.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:52.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:52.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:52.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:53.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:53.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:53.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:53.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:54.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:54.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:54.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:54.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:55.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:55.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:55.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:55.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:55.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:22:55.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:55.733596+0000) 2019-10-18T03:22:56.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:56.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:56.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:56.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:56.355 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:22:56.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:56.352478+0000) 2019-10-18T03:22:56.699 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:22:56.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:56.699545+0000) 2019-10-18T03:22:56.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:22:56.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:22:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:57.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:57.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:58.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:58.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:58.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:58.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:59.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:59.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:22:59.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:22:59.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:22:59.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:22:59.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:22:59.349174+0000) 2019-10-18T03:23:00.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:00.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:00.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:00.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:01.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:01.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:01.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:01.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:01.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:01.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:02.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:02.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:02.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:02.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:03.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:03.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:03.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:03.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:04.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:04.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:04.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:04.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:05.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:05.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:05.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:05.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:05.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:23:05.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:05.733690+0000) 2019-10-18T03:23:06.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:06.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:06.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:06.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:06.352 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:23:06.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:06.352690+0000) 2019-10-18T03:23:06.699 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:06.695+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:06.699661+0000) 2019-10-18T03:23:06.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:06.723+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:07.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:07.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:07.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:07.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:08.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:08.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:08.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:08.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:09.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:09.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:09.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:09.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:09.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:23:09.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:09.349271+0000) 2019-10-18T03:23:10.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:10.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:10.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:10.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:11.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:11.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:11.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:11.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:11.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:11.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:12.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:12.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:12.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:12.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:13.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:13.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:13.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:13.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:14.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:14.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:14.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:14.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:15.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:15.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:15.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:15.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:15.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:23:15.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:15.733783+0000) 2019-10-18T03:23:16.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:16.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:16.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:16.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:16.352 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:23:16.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:16.352902+0000) 2019-10-18T03:23:16.700 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:16.694+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:16.699787+0000) 2019-10-18T03:23:16.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:16.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:17.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:17.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:17.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:17.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:18.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:18.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:18.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:18.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:19.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:19.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:19.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:19.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:19.348 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:23:19.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:19.349387+0000) 2019-10-18T03:23:20.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:20.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:20.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:20.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:21.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:21.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:21.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:21.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:21.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:21.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:22.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:22.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:22.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:22.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:23.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:23.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:23.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:23.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:24.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:24.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:24.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:24.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:25.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:25.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:25.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:25.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:25.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:23:25.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:25.733877+0000) 2019-10-18T03:23:26.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:26.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:26.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:26.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:26.352 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:23:26.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:26.353110+0000) 2019-10-18T03:23:26.699 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:26.694+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:26.699920+0000) 2019-10-18T03:23:26.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:26.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:27.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:27.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:27.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:27.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:28.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:28.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:28.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:28.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:29.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:29.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:29.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:29.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:29.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:23:29.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:29.349501+0000) 2019-10-18T03:23:30.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:30.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:30.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:30.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:31.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:31.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:31.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:31.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:31.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:31.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:32.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:32.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:33.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:33.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:33.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:33.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:34.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:34.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:34.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:34.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:35.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:35.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:35.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:35.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:35.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:23:35.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:35.733974+0000) 2019-10-18T03:23:36.352 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:23:36.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:36.353294+0000) 2019-10-18T03:23:36.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:36.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:36.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:36.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:36.699 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:36.694+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:36.700031+0000) 2019-10-18T03:23:36.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:36.726+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:37.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:37.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:37.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:37.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:38.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:38.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:38.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:38.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:39.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:23:39.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:39.349612+0000) 2019-10-18T03:23:39.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:39.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:39.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:39.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:40.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:40.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:40.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:40.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:41.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:41.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:41.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:41.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:41.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:41.730+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:42.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:42.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:42.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:42.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:43.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:43.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:43.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:43.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:44.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:44.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:44.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:44.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:45.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:45.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:45.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:45.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:45.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:23:45.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:45.734070+0000) 2019-10-18T03:23:46.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:23:46.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:46.353502+0000) 2019-10-18T03:23:46.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:46.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:46.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:46.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:46.700 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:46.693+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:46.700147+0000) 2019-10-18T03:23:46.700 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:46.693+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:46.701083+0000) 2019-10-18T03:23:46.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:46.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:47.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:47.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:47.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:47.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:48.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:48.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:48.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:48.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:49.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:23:49.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:49.349720+0000) 2019-10-18T03:23:49.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:49.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:49.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:49.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:50.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:50.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:50.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:50.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:51.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:23:51.193+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:51.200352+0000) 2019-10-18T03:23:51.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:51.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:51.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:51.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:51.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:51.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:52.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:52.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:52.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:52.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:53.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:53.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:53.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:53.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:54.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:54.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:54.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:54.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:55.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:55.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:55.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:55.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:55.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:23:55.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:55.734167+0000) 2019-10-18T03:23:56.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:23:56.348+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:56.353666+0000) 2019-10-18T03:23:56.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:56.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:56.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:56.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:56.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:23:56.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:23:57.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:57.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:57.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:57.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:58.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:58.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:58.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:58.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:23:59.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:23:59.344+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:23:59.349835+0000) 2019-10-18T03:23:59.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:59.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:23:59.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:23:59.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:00.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:00.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:00.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:00.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:01.199 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:24:01.193+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:01.200455+0000) 2019-10-18T03:24:01.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:01.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:01.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:01.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:01.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:01.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:02.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:02.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:02.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:02.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:03.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:03.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:03.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:03.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:04.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:04.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:04.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:04.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:05.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:05.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:05.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:05.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:05.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:24:05.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:05.734261+0000) 2019-10-18T03:24:06.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:24:06.347+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:06.353849+0000) 2019-10-18T03:24:06.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:06.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:06.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:06.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:06.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:06.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:07.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:07.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:07.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:07.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:08.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:08.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:08.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:08.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:09.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:24:09.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:09.349947+0000) 2019-10-18T03:24:09.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:09.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:09.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:09.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:10.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:10.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:10.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:10.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:11.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:24:11.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:11.200555+0000) 2019-10-18T03:24:11.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:11.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:11.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:11.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:11.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:11.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:12.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:12.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:12.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:12.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:13.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:13.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:13.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:13.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:14.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:14.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:14.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:14.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:15.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:15.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:15.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:15.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:15.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:24:15.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:15.734359+0000) 2019-10-18T03:24:16.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:24:16.347+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:16.353988+0000) 2019-10-18T03:24:16.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:16.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:16.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:16.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:16.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:16.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:17.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:17.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:17.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:17.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:18.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:18.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:18.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:18.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:19.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:24:19.343+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:19.350066+0000) 2019-10-18T03:24:19.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:19.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:19.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:19.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:20.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:20.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:20.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:20.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:21.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:24:21.196+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:21.200659+0000) 2019-10-18T03:24:21.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:21.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:21.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:21.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:21.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:21.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:22.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:22.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:22.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:22.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:23.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:23.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:23.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:23.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:24.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:24.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:24.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:24.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:25.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:25.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:25.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:25.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:25.733 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:24:25.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:25.734469+0000) 2019-10-18T03:24:26.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:24:26.351+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:26.354217+0000) 2019-10-18T03:24:26.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:26.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:26.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:26.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:26.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:26.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:27.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:27.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:27.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:27.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:28.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:28.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:28.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:28.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:29.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:24:29.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:29.350162+0000) 2019-10-18T03:24:29.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:29.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:29.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:29.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:30.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:30.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:31.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:24:31.196+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:31.200774+0000) 2019-10-18T03:24:31.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:31.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:31.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:31.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:31.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:31.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:32.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:32.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:32.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:32.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:33.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:33.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:33.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:33.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:34.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:34.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:34.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:34.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:35.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:35.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:35.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:35.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:35.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:24:35.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:35.734564+0000) 2019-10-18T03:24:36.353 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:24:36.351+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:36.354399+0000) 2019-10-18T03:24:36.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:36.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:36.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:36.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:36.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:36.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:37.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:37.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:37.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:37.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:38.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:38.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:38.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:38.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:39.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:24:39.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:39.350283+0000) 2019-10-18T03:24:39.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:39.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:39.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:39.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:40.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:40.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:40.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:40.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:41.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:24:41.196+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:41.200875+0000) 2019-10-18T03:24:41.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:41.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:41.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:41.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:41.743 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:41.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:42.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:42.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:42.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:42.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:43.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:43.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:43.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:43.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:44.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:44.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:44.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:44.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:45.734 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:24:45.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:45.734659+0000) 2019-10-18T03:24:45.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:45.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:45.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:45.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:46.354 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:24:46.351+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:46.354582+0000) 2019-10-18T03:24:46.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:46.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:46.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:46.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:46.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:46.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:47.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:47.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:47.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:47.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:48.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:48.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:48.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:48.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:49.349 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:24:49.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:49.350376+0000) 2019-10-18T03:24:49.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:49.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:49.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:49.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:50.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:50.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:50.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:50.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:51.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:24:51.196+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:51.200975+0000) 2019-10-18T03:24:51.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:51.744+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:51.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:51.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:51.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:51.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:52.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:52.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:52.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:52.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:53.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:53.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:53.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:53.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:54.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:54.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:54.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:54.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:55.734 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:24:55.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:55.734757+0000) 2019-10-18T03:24:55.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:55.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:55.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:55.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:56.354 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:24:56.351+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:56.354788+0000) 2019-10-18T03:24:56.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:24:56.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:24:56.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:56.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:56.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:56.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:58.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:57.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:58.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:57.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:59.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:59.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:59.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:59.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:24:59.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:24:59.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:24:59.350486+0000) 2019-10-18T03:24:59.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:59.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:24:59.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:24:59.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:00.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:00.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:00.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:00.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:01.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:25:01.195+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:01.201081+0000) 2019-10-18T03:25:01.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:01.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:02.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:02.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:02.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:02.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:03.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:03.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:03.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:03.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 348 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:04.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:04.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:04.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:04.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:05.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:05.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:05.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:05.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:05.734 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:25:05.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:05.734857+0000) 2019-10-18T03:25:06.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:06.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:06.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:06.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:06.354 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:25:06.351+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:06.354965+0000) 2019-10-18T03:25:06.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:06.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:07.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:07.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:07.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:07.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:08.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:08.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:08.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:08.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:09.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:09.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:09.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:09.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:09.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:25:09.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:09.350583+0000) 2019-10-18T03:25:10.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:10.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:10.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:10.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:11.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:11.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:11.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:11.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:11.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:25:11.195+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:11.201181+0000) 2019-10-18T03:25:11.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:11.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:12.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:12.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:12.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:12.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:13.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:13.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:13.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:13.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:14.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:14.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:14.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:14.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:15.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:15.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:15.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:15.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:15.734 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:25:15.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:15.734952+0000) 2019-10-18T03:25:16.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:16.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:16.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:16.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:16.354 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:25:16.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:16.355141+0000) 2019-10-18T03:25:16.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:16.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:17.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:17.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:17.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:17.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:18.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:18.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:18.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:18.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:19.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:19.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:19.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:19.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:19.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:25:19.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:19.350705+0000) 2019-10-18T03:25:20.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:20.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:20.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:20.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:21.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:21.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:21.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:21.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:21.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:25:21.195+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:21.201281+0000) 2019-10-18T03:25:21.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:21.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:22.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:22.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:22.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:22.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:23.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:22.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:23.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:22.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:23.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:23.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:23.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:23.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:24.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:24.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:24.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:24.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:25.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:25:25.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:25.735047+0000) 2019-10-18T03:25:25.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:25.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:25.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:25.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:26.354 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:25:26.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:26.355327+0000) 2019-10-18T03:25:26.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:26.747+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:26.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:26.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:26.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:26.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:27.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:27.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:29.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:28.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:29.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:28.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:29.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:25:29.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:29.350817+0000) 2019-10-18T03:25:30.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:30.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:30.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:30.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:31.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:31.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:31.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:31.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:31.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:25:31.194+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:31.201384+0000) 2019-10-18T03:25:31.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:31.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:32.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:32.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:32.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:32.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:33.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:33.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:33.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:33.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:34.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:34.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:34.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:34.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:35.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:35.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:35.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:35.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:35.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:25:35.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:35.735146+0000) 2019-10-18T03:25:36.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:36.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:36.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:36.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:36.354 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:25:36.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:36.355506+0000) 2019-10-18T03:25:36.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:36.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:37.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:37.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:37.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:37.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:38.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:38.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:38.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:39.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:39.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:39.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:39.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:39.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:25:39.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:39.350930+0000) 2019-10-18T03:25:40.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:40.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:40.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:40.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:41.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:41.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:41.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:41.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:41.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:25:41.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:41.201483+0000) 2019-10-18T03:25:41.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:41.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:42.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:42.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:42.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:42.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:42.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:42.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:42.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:42.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:43.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:43.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:43.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:43.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:44.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:44.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:44.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:44.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:45.734 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:25:45.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:45.735248+0000) 2019-10-18T03:25:45.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:45.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:45.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:45.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:46.355 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:25:46.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:46.355697+0000) 2019-10-18T03:25:46.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:46.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:46.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:46.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:46.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:46.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:47.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:47.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:47.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:47.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:48.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:48.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:48.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:48.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:49.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:25:49.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:49.351042+0000) 2019-10-18T03:25:49.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:49.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:49.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:49.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:50.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:50.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:50.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:50.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:51.200 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:25:51.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:51.201582+0000) 2019-10-18T03:25:51.755 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:51.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:51.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:51.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:51.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:51.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:52.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:52.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:53.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:53.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:53.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:53.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:54.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:54.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:54.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:54.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:55.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:25:55.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:55.735360+0000) 2019-10-18T03:25:55.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:55.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:55.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:55.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:56.355 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:25:56.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:56.355909+0000) 2019-10-18T03:25:56.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:56.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:56.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:56.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:56.755 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:25:56.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:25:57.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:57.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:57.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:57.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:58.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:58.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:58.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:58.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:25:59.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:25:59.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:25:59.351155+0000) 2019-10-18T03:25:59.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:59.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:25:59.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:25:59.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:00.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:00.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:00.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:00.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:01.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:26:01.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:01.201684+0000) 2019-10-18T03:26:01.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:01.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:01.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:01.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:01.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:01.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:02.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:02.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:02.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:02.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:03.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:03.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:03.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:03.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:04.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:04.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:04.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:04.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:05.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:05.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:05.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:05.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:05.734 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:26:05.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:05.735454+0000) 2019-10-18T03:26:06.355 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:26:06.350+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:06.356093+0000) 2019-10-18T03:26:06.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:06.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:06.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:06.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:06.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:06.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:07.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:07.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:07.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:07.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:08.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:08.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:08.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:08.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:09.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:26:09.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:09.351265+0000) 2019-10-18T03:26:09.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:09.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:09.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:09.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:10.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:10.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:10.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:10.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:11.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:26:11.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:11.201783+0000) 2019-10-18T03:26:11.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:11.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:11.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:11.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:11.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:11.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:12.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:12.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:12.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:12.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:13.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:13.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:13.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:13.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:14.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:14.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:14.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:14.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:15.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:15.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:15.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:15.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:15.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:26:15.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:15.735550+0000) 2019-10-18T03:26:16.355 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:26:16.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:16.356282+0000) 2019-10-18T03:26:16.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:16.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:16.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:16.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:16.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:16.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:17.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:17.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:17.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:17.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:18.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:18.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:18.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:18.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:19.350 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:26:19.350+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:19.351380+0000) 2019-10-18T03:26:19.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:19.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:19.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:19.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:20.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:20.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:20.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:20.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:21.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:26:21.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:21.201890+0000) 2019-10-18T03:26:21.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:21.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:21.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:21.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:21.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:21.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:22.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:22.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:22.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:22.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:23.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:23.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:23.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:23.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:24.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:24.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:24.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:24.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:25.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:25.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:25.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:25.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:25.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:26:25.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:25.735653+0000) 2019-10-18T03:26:26.355 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:26:26.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:26.356468+0000) 2019-10-18T03:26:26.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:26.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:26.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:26.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:26.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:26.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:27.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:27.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:27.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:27.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:28.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:28.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:28.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:28.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:29.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:26:29.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:29.351493+0000) 2019-10-18T03:26:29.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:29.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:29.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:29.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:30.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:30.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:30.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:30.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:31.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:26:31.201+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:31.201992+0000) 2019-10-18T03:26:31.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:31.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:31.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:31.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:31.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:31.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:32.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:32.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:32.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:32.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:33.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:33.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:33.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:33.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:34.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:34.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:34.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:34.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:35.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:35.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:35.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:35.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:35.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:26:35.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:35.735752+0000) 2019-10-18T03:26:36.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:26:36.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:36.356647+0000) 2019-10-18T03:26:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:36.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:36.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:36.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:36.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:37.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:37.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:37.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:37.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:38.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:38.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:38.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:38.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:39.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:26:39.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:39.351607+0000) 2019-10-18T03:26:39.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:39.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:39.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:39.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:40.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:40.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:40.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:40.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:41.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:26:41.201+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:41.202092+0000) 2019-10-18T03:26:41.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:41.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:41.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:41.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:41.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:41.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:42.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:42.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:42.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:42.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:43.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:43.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:43.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:43.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:44.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:44.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:44.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:44.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:45.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:45.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:45.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:45.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:45.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:26:45.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:45.735848+0000) 2019-10-18T03:26:46.356 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:26:46.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:46.356829+0000) 2019-10-18T03:26:46.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:46.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:46.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:46.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:46.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:46.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:47.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:47.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:47.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:47.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:48.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:48.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:48.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:48.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:49.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:26:49.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:49.351720+0000) 2019-10-18T03:26:49.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:49.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:49.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:49.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:50.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:50.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:50.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:50.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:51.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:26:51.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:51.202194+0000) 2019-10-18T03:26:51.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:51.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:51.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:51.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:51.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:51.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:52.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:52.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:52.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:52.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:53.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:53.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:53.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:53.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:54.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:54.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:54.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:54.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:55.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:55.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:55.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:55.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:55.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:26:55.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:55.735949+0000) 2019-10-18T03:26:56.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:56.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:56.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:56.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:56.356 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:26:56.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:56.357048+0000) 2019-10-18T03:26:56.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:26:56.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:26:57.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:57.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:57.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:57.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:58.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:58.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:58.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:58.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:59.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:59.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:26:59.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:26:59.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:26:59.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:26:59.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:26:59.351835+0000) 2019-10-18T03:27:00.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:00.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:00.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:00.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:01.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:27:01.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:01.202301+0000) 2019-10-18T03:27:01.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:01.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:01.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:01.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:01.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:01.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:02.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:02.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:02.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:02.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:03.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:03.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:03.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:03.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:04.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:04.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:04.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:04.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:05.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:05.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:05.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:05.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:05.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:27:05.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:05.736048+0000) 2019-10-18T03:27:06.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:06.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:06.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:06.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:06.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:27:06.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:06.357227+0000) 2019-10-18T03:27:06.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:06.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:07.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:07.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:07.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:07.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:08.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:08.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:08.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:08.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:09.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:09.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:09.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:09.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:09.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:27:09.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:09.351948+0000) 2019-10-18T03:27:10.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:10.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:10.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:10.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:11.201 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:27:11.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:11.202402+0000) 2019-10-18T03:27:11.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:11.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:11.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:11.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:11.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:11.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:12.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:12.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:12.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:12.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:13.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:13.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:13.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:13.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:14.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:14.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:14.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:14.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:15.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:15.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:15.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:15.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:15.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:27:15.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:15.736147+0000) 2019-10-18T03:27:16.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:16.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:16.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:16.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:16.356 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:27:16.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:16.357410+0000) 2019-10-18T03:27:16.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:16.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:17.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:17.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:17.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:17.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:18.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:18.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:18.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:18.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:19.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:19.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:19.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:19.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:19.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:27:19.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:19.352058+0000) 2019-10-18T03:27:20.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:20.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:20.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:20.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:21.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:27:21.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:21.202502+0000) 2019-10-18T03:27:21.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:21.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:21.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:21.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:21.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:21.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:22.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:22.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:22.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:22.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:23.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:23.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:23.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:23.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:24.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:24.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:24.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:24.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:25.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:25.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:25.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:25.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:25.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:27:25.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:25.736246+0000) 2019-10-18T03:27:26.357 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:27:26.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:26.357623+0000) 2019-10-18T03:27:26.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:26.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:26.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:26.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:26.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:26.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:27.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:27.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:27.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:27.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:28.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:28.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:28.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:28.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:29.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:27:29.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:29.352174+0000) 2019-10-18T03:27:29.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:29.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:29.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:29.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:30.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:30.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:30.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:30.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:31.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:27:31.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:31.202600+0000) 2019-10-18T03:27:31.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:31.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:31.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:31.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:31.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:31.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:32.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:32.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:32.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:32.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:33.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:33.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:33.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:33.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:34.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:34.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:34.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:34.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:35.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:35.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:35.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:35.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:35.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:27:35.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:35.736344+0000) 2019-10-18T03:27:36.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:27:36.353+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:36.357834+0000) 2019-10-18T03:27:36.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:36.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:36.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:36.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:36.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:36.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:37.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:37.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:37.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:37.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:38.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:38.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:38.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:38.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:39.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:39.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:39.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:39.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:39.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:27:39.349+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:39.352287+0000) 2019-10-18T03:27:40.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:40.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:40.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:40.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:41.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:27:41.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:41.202700+0000) 2019-10-18T03:27:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:41.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:41.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:41.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:41.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:42.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:42.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:42.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:42.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:43.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:43.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:43.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:43.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:44.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:44.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:44.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:44.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:45.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:45.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:45.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:45.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:45.735 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:27:45.731+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:45.736440+0000) 2019-10-18T03:27:46.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:46.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:46.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:46.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:46.357 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:27:46.352+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:46.358016+0000) 2019-10-18T03:27:46.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:46.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:47.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:47.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:47.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:47.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:48.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:48.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:48.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:48.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:49.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:49.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:49.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:49.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:49.351 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:27:49.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:49.352401+0000) 2019-10-18T03:27:50.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:50.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:50.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:50.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:51.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:51.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:51.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:51.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:51.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:27:51.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:51.202802+0000) 2019-10-18T03:27:51.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:51.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:52.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:52.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:52.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:52.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:53.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:53.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:53.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:53.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:54.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:54.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:54.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:54.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:55.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:55.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:55.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:55.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:55.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:27:55.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:55.736537+0000) 2019-10-18T03:27:56.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:56.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:56.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:56.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:56.364 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:27:56.356+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:56.358205+0000) 2019-10-18T03:27:56.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:27:56.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:27:57.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:57.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:57.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:57.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:58.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:58.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:58.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:58.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:59.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:59.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:27:59.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:27:59.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:27:59.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:27:59.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:27:59.352514+0000) 2019-10-18T03:28:00.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:00.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:00.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:00.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:01.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:28:01.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:01.202905+0000) 2019-10-18T03:28:01.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:01.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:01.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:01.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:01.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:01.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:02.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:02.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:02.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:02.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:03.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:03.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:03.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:03.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:04.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:04.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:04.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:04.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:05.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:05.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:05.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:05.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:05.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:28:05.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:05.736635+0000) 2019-10-18T03:28:06.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:06.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:06.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:06.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:06.357 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:28:06.356+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:06.358382+0000) 2019-10-18T03:28:06.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:06.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:07.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:07.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:07.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:07.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:08.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:08.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:08.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:08.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:09.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:09.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:09.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:09.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:09.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:28:09.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:09.352621+0000) 2019-10-18T03:28:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:10.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:10.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:11.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:11.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:11.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:11.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:11.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:28:11.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:11.203003+0000) 2019-10-18T03:28:11.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:11.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:12.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:12.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:12.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:12.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:13.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:13.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:13.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:13.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:14.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:14.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:14.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:14.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:15.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:15.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:15.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:15.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:15.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:28:15.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:15.736733+0000) 2019-10-18T03:28:15.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:15.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:15.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:15.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:16.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:28:16.356+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:16.358563+0000) 2019-10-18T03:28:16.779 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:16.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:17.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:17.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:17.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:17.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:18.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:18.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:18.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:18.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:19.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:19.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:19.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:19.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:19.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:28:19.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:19.352733+0000) 2019-10-18T03:28:20.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:20.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:20.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:20.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:21.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:21.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:21.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:21.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:21.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:28:21.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:21.203103+0000) 2019-10-18T03:28:21.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:21.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:22.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:22.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:22.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:22.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:23.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:23.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:23.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:23.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:24.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:24.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:24.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:24.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:25.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:25.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:25.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:25.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:25.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:28:25.730+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:25.736837+0000) 2019-10-18T03:28:26.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:26.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:26.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:26.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:26.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:28:26.356+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:26.358713+0000) 2019-10-18T03:28:26.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:26.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:27.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:27.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:27.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:27.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:28.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:28.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:28.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:28.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:29.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:29.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:29.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:29.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:29.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:28:29.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:29.352846+0000) 2019-10-18T03:28:30.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:30.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:30.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:30.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:31.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:31.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:31.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:31.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:31.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:28:31.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:31.203206+0000) 2019-10-18T03:28:31.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:31.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:32.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:32.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:32.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:32.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:33.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:33.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:33.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:33.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:34.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:34.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:34.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:34.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:35.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:35.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:35.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:35.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:35.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:28:35.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:35.736935+0000) 2019-10-18T03:28:36.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:36.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:36.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:36.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:36.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:28:36.356+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:36.358922+0000) 2019-10-18T03:28:36.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:36.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:37.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:37.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:37.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:37.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:38.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:38.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:38.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:38.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:39.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:39.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:39.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:39.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:39.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:28:39.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:39.352971+0000) 2019-10-18T03:28:40.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:40.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:40.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:40.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:40.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:40.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:40.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:40.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:41.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:28:41.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:41.203303+0000) 2019-10-18T03:28:41.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:41.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:41.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:41.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:41.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:41.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:42.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:42.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:42.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:42.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:43.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:43.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:43.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:43.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:45.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:44.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:45.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:44.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:45.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:28:45.729+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:45.737030+0000) 2019-10-18T03:28:45.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:45.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:45.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:45.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:46.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:28:46.356+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:46.359103+0000) 2019-10-18T03:28:46.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:46.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:46.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:46.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:46.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:46.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:47.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:47.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:47.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:47.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:48.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:48.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:48.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:48.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:49.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:28:49.348+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:49.353081+0000) 2019-10-18T03:28:49.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:49.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:49.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:49.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:50.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:50.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:50.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:50.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:51.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:28:51.201+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:51.203407+0000) 2019-10-18T03:28:51.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:51.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:51.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:51.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:51.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:51.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:52.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:52.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:52.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:52.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:53.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:53.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:53.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:53.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:54.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:54.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:54.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:54.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:55.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:28:55.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:55.737129+0000) 2019-10-18T03:28:55.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:55.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:55.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:55.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:56.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:28:56.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:56.359316+0000) 2019-10-18T03:28:56.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:28:56.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:28:56.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:56.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:56.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:56.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:57.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:57.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:57.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:57.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:58.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:58.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:58.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:58.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:28:59.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:28:59.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:28:59.353194+0000) 2019-10-18T03:28:59.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:59.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:28:59.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:28:59.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:00.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:00.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:00.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:00.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:01.202 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:29:01.201+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:01.203507+0000) 2019-10-18T03:29:01.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:01.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:01.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:01.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:01.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:01.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:02.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:02.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:02.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:02.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:03.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:03.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:03.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:03.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:04.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:04.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:05.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:29:05.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:05.737228+0000) 2019-10-18T03:29:05.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:05.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:05.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:05.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:06.358 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:29:06.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:06.359505+0000) 2019-10-18T03:29:06.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:06.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:06.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:06.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:06.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:06.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:07.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:07.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:07.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:07.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:08.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:08.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:08.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:08.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:09.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:29:09.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:09.353301+0000) 2019-10-18T03:29:09.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:09.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:09.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:09.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:10.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:10.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:10.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:10.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:11.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:29:11.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:11.203603+0000) 2019-10-18T03:29:11.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:11.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:11.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:11.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:11.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:11.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:12.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:12.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:12.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:12.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:13.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:13.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:13.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:13.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:14.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:14.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:14.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:14.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:15.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:29:15.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:15.737323+0000) 2019-10-18T03:29:15.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:15.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:15.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:15.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:16.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:29:16.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:16.359691+0000) 2019-10-18T03:29:16.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:16.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:16.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:16.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:16.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:16.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:17.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:17.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:17.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:17.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:18.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:18.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:18.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:18.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:19.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:29:19.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:19.353399+0000) 2019-10-18T03:29:19.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:19.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:19.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:19.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:20.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:20.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:21.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:29:21.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:21.203701+0000) 2019-10-18T03:29:21.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:21.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:21.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:21.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:21.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:21.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:22.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:22.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:22.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:22.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:23.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:23.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:23.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:23.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:24.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:24.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:24.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:24.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:25.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:29:25.732+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:25.737422+0000) 2019-10-18T03:29:25.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:25.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:25.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:25.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:26.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:29:26.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:26.359899+0000) 2019-10-18T03:29:26.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:26.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:26.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:26.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:26.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:26.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:27.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:27.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:27.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:27.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:28.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:28.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:28.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:28.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:29.352 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:29:29.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:29.353512+0000) 2019-10-18T03:29:29.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:29.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:29.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:29.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:30.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:30.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:30.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:30.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:31.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:29:31.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:31.203797+0000) 2019-10-18T03:29:31.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:31.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:31.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:31.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:31.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:31.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:32.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:32.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:32.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:32.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:33.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:33.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:34.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:34.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:34.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:34.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:35.736 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:29:35.736+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:35.737527+0000) 2019-10-18T03:29:35.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:35.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:35.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:35.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:36.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:29:36.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:36.360083+0000) 2019-10-18T03:29:36.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:36.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:36.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:36.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:36.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:36.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:37.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:37.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:37.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:37.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:38.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:38.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:38.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:38.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:39.357 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:29:39.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:39.353636+0000) 2019-10-18T03:29:39.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:39.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:39.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:39.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:40.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:40.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:40.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:40.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:41.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:29:41.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:41.203893+0000) 2019-10-18T03:29:41.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:41.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:41.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:41.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:41.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:41.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:42.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:42.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:42.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:42.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:43.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:43.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:43.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:43.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:44.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:44.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:44.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:44.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:45.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:29:45.735+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:45.737622+0000) 2019-10-18T03:29:45.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:45.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:45.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:45.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:46.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:29:46.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:46.360294+0000) 2019-10-18T03:29:46.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:46.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:46.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:46.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:46.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:46.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:47.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:47.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:47.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:47.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:48.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:48.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:48.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:48.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:49.362 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:29:49.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:49.353748+0000) 2019-10-18T03:29:49.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:49.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:49.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:49.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:50.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:50.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:50.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:50.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:51.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:29:51.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:51.204002+0000) 2019-10-18T03:29:51.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:51.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:51.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:51.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:51.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:51.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:52.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:52.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:52.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:52.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:53.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:53.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:53.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:53.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:54.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:54.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:54.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:54.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:55.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:29:55.735+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:55.737718+0000) 2019-10-18T03:29:55.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:55.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:55.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:55.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:56.359 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:29:56.355+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:56.360498+0000) 2019-10-18T03:29:56.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:29:56.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:29:56.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:56.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:56.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:56.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:57.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:57.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:57.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:57.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:58.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:58.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:58.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:58.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:29:59.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:29:59.347+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:29:59.353862+0000) 2019-10-18T03:29:59.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:59.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:29:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:29:59.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:00.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:00.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:01.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:30:01.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:01.204108+0000) 2019-10-18T03:30:01.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:01.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:01.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:01.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:01.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:01.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:02.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:02.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:03.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:02.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:03.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:03.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:03.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:03.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:04.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:04.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:04.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:04.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:05.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:30:05.735+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:05.737814+0000) 2019-10-18T03:30:05.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:05.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:05.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:05.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:06.360 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:30:06.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:06.360683+0000) 2019-10-18T03:30:06.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:06.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:06.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:06.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:06.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:06.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:07.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:07.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:07.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:07.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:08.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:08.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:08.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:08.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:09.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:30:09.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:09.353971+0000) 2019-10-18T03:30:09.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:09.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:09.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:09.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:10.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:10.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:10.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:10.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:11.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:30:11.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:11.204210+0000) 2019-10-18T03:30:11.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:11.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:11.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:11.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:11.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:11.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:12.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:12.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:12.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:12.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:13.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:13.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:13.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:13.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:14.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:14.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:14.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:14.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:15.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:30:15.734+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:15.737909+0000) 2019-10-18T03:30:15.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:15.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:15.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:15.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:16.360 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:30:16.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:16.360863+0000) 2019-10-18T03:30:16.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:16.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:16.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:16.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:16.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:16.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:17.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:17.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:17.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:17.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:18.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:18.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:18.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:18.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:19.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:30:19.346+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:19.354086+0000) 2019-10-18T03:30:19.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:19.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:19.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:19.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:20.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:20.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:20.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:20.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:21.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:30:21.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:21.204317+0000) 2019-10-18T03:30:21.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:21.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:21.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:21.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:21.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:21.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:22.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:22.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:22.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:22.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:23.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:23.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:23.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:23.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:24.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:24.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:24.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:24.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:25.745 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:30:25.734+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:25.738006+0000) 2019-10-18T03:30:25.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:25.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:25.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:25.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:26.360 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:30:26.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:26.361043+0000) 2019-10-18T03:30:26.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:26.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:26.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:26.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:26.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:26.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:27.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:27.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:27.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:27.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:28.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:28.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:28.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:28.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:29.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:30:29.350+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:29.354212+0000) 2019-10-18T03:30:29.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:29.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:29.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:29.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:30.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:30.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:30.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:30.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:31.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:30:31.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:31.204423+0000) 2019-10-18T03:30:31.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:31.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:31.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:31.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:31.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:31.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:32.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:32.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:32.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:32.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:33.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:33.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:33.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:33.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:34.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:34.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:34.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:34.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:35.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:30:35.734+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:35.738116+0000) 2019-10-18T03:30:35.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:35.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:35.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:35.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:36.360 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:30:36.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:36.361190+0000) 2019-10-18T03:30:36.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:36.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:36.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:36.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:36.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:36.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:37.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:37.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:37.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:37.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:38.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:38.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:38.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:38.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:39.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:30:39.350+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:39.354311+0000) 2019-10-18T03:30:39.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:39.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:39.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:39.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:40.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:40.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:40.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:40.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:41.203 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:30:41.198+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:41.204519+0000) 2019-10-18T03:30:41.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:41.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:41.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:41.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:41.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:41.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:42.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:42.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:42.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:42.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:43.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:43.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:43.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:43.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:44.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:44.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:44.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:44.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:45.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:30:45.734+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:45.738212+0000) 2019-10-18T03:30:45.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:45.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:45.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:45.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:46.361 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:30:46.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:46.361397+0000) 2019-10-18T03:30:46.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:46.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:46.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:46.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:46.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:46.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:47.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:47.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:48.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:47.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:49.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:49.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:49.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:49.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:49.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:30:49.350+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:49.354411+0000) 2019-10-18T03:30:49.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:49.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:49.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:49.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:50.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:50.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:50.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:50.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:51.205 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:30:51.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:51.204623+0000) 2019-10-18T03:30:51.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:51.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:51.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:51.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:51.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:51.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:52.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:52.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:52.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:52.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:53.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:53.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:53.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:53.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:54.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:54.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:54.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:54.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:55.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:30:55.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:55.738306+0000) 2019-10-18T03:30:55.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:55.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:55.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:55.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:56.360 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:30:56.354+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:56.361577+0000) 2019-10-18T03:30:56.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:30:56.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:30:57.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:57.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:57.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:57.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:58.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:58.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:58.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:58.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:59.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:59.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:59.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:59.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:30:59.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:30:59.350+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:30:59.354509+0000) 2019-10-18T03:30:59.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:59.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:30:59.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:30:59.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:00.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:00.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:00.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:00.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:01.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:31:01.197+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:01.204731+0000) 2019-10-18T03:31:01.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:01.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:01.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:01.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:01.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:01.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:02.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:02.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:02.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:02.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:03.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:03.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:03.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:03.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:04.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:04.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:04.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:04.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:05.740 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:31:05.733+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:05.738406+0000) 2019-10-18T03:31:05.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:05.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:05.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:05.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:06.361 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:31:06.358+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:06.361766+0000) 2019-10-18T03:31:06.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:06.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:06.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:06.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:06.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:06.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:07.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:07.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:07.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:07.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:08.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:08.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:08.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:08.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:09.353 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:31:09.354+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:09.354604+0000) 2019-10-18T03:31:09.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:09.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:09.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:09.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:10.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:10.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:10.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:10.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:11.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:31:11.201+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:11.204830+0000) 2019-10-18T03:31:11.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:11.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:11.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:11.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:11.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:11.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:13.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:13.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:13.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:13.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:14.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:14.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:14.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:14.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:15.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:15.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:15.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:15.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:15.737 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:31:15.737+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:15.738500+0000) 2019-10-18T03:31:16.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:16.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:16.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:16.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:16.361 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:31:16.358+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:16.361952+0000) 2019-10-18T03:31:16.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:16.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:17.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:17.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:17.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:17.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:18.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:18.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:18.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:18.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:19.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:19.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:19.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:19.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:19.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:31:19.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:19.354716+0000) 2019-10-18T03:31:20.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:20.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:20.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:20.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:21.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:21.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:21.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:21.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:21.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:31:21.201+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:21.204935+0000) 2019-10-18T03:31:21.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:21.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:22.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:22.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:23.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:23.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:23.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:23.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:24.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:24.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:24.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:24.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:25.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:25.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:25.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:25.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:25.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:31:25.736+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:25.738594+0000) 2019-10-18T03:31:26.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:26.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:26.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:26.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:26.361 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:31:26.357+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:26.362135+0000) 2019-10-18T03:31:26.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:26.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:27.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:27.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:27.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:27.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:28.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:28.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:28.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:28.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:29.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:29.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:29.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:29.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:29.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:31:29.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:29.354825+0000) 2019-10-18T03:31:30.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:30.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:30.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:30.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:31.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:31.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:31.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:31.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:31.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:31:31.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:31.205054+0000) 2019-10-18T03:31:31.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:31.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:32.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:32.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:32.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:32.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:33.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:33.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:33.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:33.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:34.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:34.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:34.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:34.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:35.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:35.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:35.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:35.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:35.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:31:35.736+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:35.738695+0000) 2019-10-18T03:31:36.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:36.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:36.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:36.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:36.361 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:31:36.361+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:36.362369+0000) 2019-10-18T03:31:36.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:36.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:37.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:37.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:37.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:37.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:38.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:38.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:38.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:38.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:39.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:39.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:39.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:39.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:39.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:31:39.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:39.354930+0000) 2019-10-18T03:31:40.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:40.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:40.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:40.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:41.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:41.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:41.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:41.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:41.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:31:41.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:41.205157+0000) 2019-10-18T03:31:41.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:41.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:42.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:42.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:42.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:42.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:43.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:43.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:43.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:43.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:44.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:44.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:44.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:44.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:45.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:45.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:45.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:45.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:45.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:31:45.736+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:45.738792+0000) 2019-10-18T03:31:46.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:46.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:46.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:46.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:46.362 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:31:46.361+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:46.362554+0000) 2019-10-18T03:31:46.814 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:46.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:47.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:47.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:47.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:47.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:48.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:48.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:48.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:48.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:49.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:49.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:49.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:49.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:49.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:31:49.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:49.355040+0000) 2019-10-18T03:31:50.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:50.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:50.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:50.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:51.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:51.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:51.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:51.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:51.208 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:31:51.200+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:51.205263+0000) 2019-10-18T03:31:51.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:51.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:52.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:52.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:52.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:52.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:53.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:53.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:53.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:53.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:54.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:54.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:54.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:54.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:55.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:55.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:55.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:55.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:55.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:31:55.736+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:55.738890+0000) 2019-10-18T03:31:56.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:56.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:56.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:56.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:56.362 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:31:56.361+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:56.362732+0000) 2019-10-18T03:31:56.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:31:56.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:31:57.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:57.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:57.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:57.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:58.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:58.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:59.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:59.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:31:59.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:31:59.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:31:59.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:31:59.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:31:59.355150+0000) 2019-10-18T03:32:00.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:00.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:00.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:00.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:01.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:01.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:01.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:01.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:01.204 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:01.199+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:01.205369+0000) 2019-10-18T03:32:01.816 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:01.811+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:02.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:02.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:03.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:03.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:03.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:03.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:04.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:04.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:04.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:04.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:05.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:05.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:05.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:05.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:05.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:05.735+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:05.738986+0000) 2019-10-18T03:32:06.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:05.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:06.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:06.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:06.362 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:32:06.361+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:06.362925+0000) 2019-10-18T03:32:06.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:06.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:07.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:07.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:07.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:07.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:08.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:08.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:08.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:08.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:09.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:09.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:09.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:09.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:09.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:32:09.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:09.355257+0000) 2019-10-18T03:32:10.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:10.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:10.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:10.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:11.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:11.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:11.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:11.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:11.205 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:11.203+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:11.205462+0000) 2019-10-18T03:32:11.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:11.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:12.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:12.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:12.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:12.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:13.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:13.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:13.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:13.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:14.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:14.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:14.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:14.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:15.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:15.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:15.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:15.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:15.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:15.735+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:15.739088+0000) 2019-10-18T03:32:16.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:16.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:16.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:16.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:16.362 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:32:16.361+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:16.363108+0000) 2019-10-18T03:32:16.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:16.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:17.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:17.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:21:46.183532+0000 (oldest deadline 2019-10-18T03:22:06.183532+0000) 2019-10-18T03:32:17.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:17.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:18.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:18.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:19.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:19.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:19.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:32:19.353+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:19.355363+0000) 2019-10-18T03:32:20.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:20.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:21.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:21.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:21.207 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:21.203+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:21.205566+0000) 2019-10-18T03:32:21.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:21.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:22.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:22.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:23.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:23.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:24.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:24.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:25.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:25.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:25.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:25.735+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:25.739178+0000) 2019-10-18T03:32:26.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:26.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:26.362 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:32:26.361+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:26.363289+0000) 2019-10-18T03:32:26.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:26.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:27.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:27.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:28.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:28.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:29.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:29.354 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:32:29.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:29.355474+0000) 2019-10-18T03:32:30.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:30.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:31.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:31.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:31.206 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:31.203+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:31.205663+0000) 2019-10-18T03:32:31.206 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:31.203+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:31.206522+0000) 2019-10-18T03:32:31.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:31.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:32.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:32.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:33.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:33.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:34.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:34.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:34.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:34.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:35.705 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:35.702+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:35.705854+0000) 2019-10-18T03:32:35.738 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:35.734+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:35.739272+0000) 2019-10-18T03:32:35.739 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:35.734+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:35.740047+0000) 2019-10-18T03:32:36.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:36.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:36.362 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:32:36.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:36.363476+0000) 2019-10-18T03:32:36.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:36.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:37.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:37.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:38.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:38.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:39.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:39.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:39.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:32:39.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:39.355589+0000) 2019-10-18T03:32:40.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:40.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:40.238 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:40.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:40.239469+0000) 2019-10-18T03:32:41.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:41.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:41.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:41.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:41.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:41.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:42.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:42.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:42.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:42.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:43.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:43.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:43.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:43.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:44.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:44.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:44.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:44.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:45.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:45.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:45.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:45.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:45.705 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:45.702+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:45.705949+0000) 2019-10-18T03:32:46.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:46.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:46.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:46.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:46.364 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:32:46.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:46.363654+0000) 2019-10-18T03:32:46.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:46.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:47.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:47.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:47.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:47.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:48.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:48.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:49.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:49.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:49.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:49.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:49.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:32:49.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:49.355700+0000) 2019-10-18T03:32:50.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:50.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:50.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:50.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:50.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:32:50.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:50.239572+0000) 2019-10-18T03:32:51.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:51.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:51.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:51.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:51.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:51.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:52.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:52.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:52.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:52.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:53.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:53.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:53.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:53.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:54.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:54.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:54.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:54.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:55.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:55.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:55.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:55.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:55.705 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:32:55.702+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:55.706048+0000) 2019-10-18T03:32:56.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:56.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:56.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:56.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:56.363 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:32:56.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:56.363843+0000) 2019-10-18T03:32:56.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:32:56.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:32:57.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:57.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:57.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:57.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:58.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:58.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:58.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:58.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:59.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:59.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:32:59.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:32:59.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:32:59.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:32:59.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:32:59.355809+0000) 2019-10-18T03:33:00.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:00.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:00.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:00.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:00.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:33:00.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:00.239678+0000) 2019-10-18T03:33:01.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:01.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:01.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:01.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:01.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:01.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:02.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:02.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:02.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:02.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:03.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:03.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:03.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:03.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:04.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:04.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:04.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:04.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:05.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:05.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:05.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:05.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:05.705 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:33:05.702+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:05.706144+0000) 2019-10-18T03:33:06.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:06.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:06.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:06.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:06.363 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:06.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:06.364021+0000) 2019-10-18T03:33:06.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:06.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:07.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:07.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:07.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:07.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:08.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:08.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:08.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:08.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:09.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:09.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:09.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:09.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:09.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:33:09.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:09.355921+0000) 2019-10-18T03:33:10.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:10.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:10.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:10.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:10.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:33:10.233+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:10.239787+0000) 2019-10-18T03:33:11.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:11.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:11.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:11.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:11.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:11.821+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:12.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:12.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:12.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:12.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:13.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:13.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:13.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:13.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:14.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:14.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:14.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:14.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:15.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:15.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:15.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:15.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:15.705 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:33:15.701+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:15.706236+0000) 2019-10-18T03:33:16.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:16.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:16.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:16.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:16.363 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:16.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:16.364205+0000) 2019-10-18T03:33:16.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:16.821+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:17.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:17.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:17.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:17.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:18.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:18.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:18.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:18.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:19.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:19.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:19.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:19.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:19.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:33:19.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:19.356033+0000) 2019-10-18T03:33:20.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:20.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:20.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:20.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:20.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:33:20.233+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:20.239888+0000) 2019-10-18T03:33:21.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:21.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:21.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:21.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:21.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:21.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:22.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:22.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:22.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:22.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:23.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:23.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:23.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:23.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:24.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:24.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:24.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:24.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:25.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:25.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:25.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:25.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:25.705 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:33:25.701+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:25.706342+0000) 2019-10-18T03:33:26.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:26.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:26.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:26.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:26.363 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:26.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:26.364375+0000) 2019-10-18T03:33:26.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:26.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:27.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:27.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:27.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:27.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:28.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:28.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:28.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:28.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:29.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:29.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:29.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:29.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:29.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:33:29.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:29.356145+0000) 2019-10-18T03:33:30.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:30.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:30.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:30.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:30.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:33:30.237+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:30.239993+0000) 2019-10-18T03:33:31.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:30.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:31.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:30.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:31.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:31.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:31.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:31.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:31.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:31.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:32.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:32.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:32.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:32.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:33.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:33.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:33.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:33.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:34.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:34.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:34.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:34.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:35.706 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:33:35.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:35.706443+0000) 2019-10-18T03:33:35.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:35.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:35.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:35.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:36.364 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:36.360+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:36.364545+0000) 2019-10-18T03:33:36.366 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:36.360+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:36.365379+0000) 2019-10-18T03:33:36.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:36.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:36.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:36.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:36.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:36.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:37.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:37.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:37.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:37.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:38.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:38.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:38.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:38.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:39.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:33:39.352+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:39.356259+0000) 2019-10-18T03:33:39.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:39.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:39.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:39.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:40.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:33:40.237+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:40.240092+0000) 2019-10-18T03:33:40.864 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:40.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:40.864941+0000) 2019-10-18T03:33:40.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:40.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:40.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:40.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:41.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:41.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:41.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:41.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:41.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:41.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:42.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:42.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:42.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:42.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:43.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:43.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:43.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:43.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:44.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:44.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:44.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:44.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:45.706 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:33:45.704+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:45.706540+0000) 2019-10-18T03:33:45.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:45.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:45.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:45.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:46.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:46.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:46.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:46.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:46.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:46.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:47.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:47.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:47.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:47.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:48.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:48.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:48.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:48.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:49.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:33:49.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:49.356374+0000) 2019-10-18T03:33:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:50.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:50.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:50.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:33:50.236+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:50.240192+0000) 2019-10-18T03:33:50.864 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:33:50.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:50.865121+0000) 2019-10-18T03:33:51.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:51.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:51.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:51.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:51.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:51.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:52.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:52.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:52.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:52.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:53.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:53.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:53.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:53.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:54.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:54.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:54.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:54.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:55.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:55.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:55.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:55.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:55.706 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:33:55.704+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:55.706635+0000) 2019-10-18T03:33:56.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:56.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:56.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:56.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:56.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:33:56.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:33:57.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:57.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:57.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:57.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:58.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:58.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:58.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:58.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:59.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:59.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:33:59.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:33:59.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:33:59.355 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:33:59.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:33:59.356486+0000) 2019-10-18T03:34:00.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:00.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:00.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:00.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:00.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:34:00.236+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:00.240298+0000) 2019-10-18T03:34:00.864 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:34:00.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:00.865304+0000) 2019-10-18T03:34:01.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:01.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:01.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:01.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:01.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:01.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:02.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:02.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:02.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:02.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:03.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:03.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:03.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:03.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:04.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:04.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:04.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:04.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:05.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:05.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:05.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:05.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:05.708 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:34:05.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:05.709542+0000) 2019-10-18T03:34:06.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:06.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:06.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:06.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:06.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:06.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:07.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:07.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:07.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:07.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:08.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:08.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:08.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:08.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:09.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:09.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:09.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:09.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:09.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:09.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:09.356601+0000) 2019-10-18T03:34:10.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:34:10.236+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:10.240399+0000) 2019-10-18T03:34:10.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:10.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:10.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:10.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:10.864 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:34:10.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:10.865484+0000) 2019-10-18T03:34:11.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:11.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:11.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:11.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:11.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:11.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:12.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:12.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:12.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:12.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:13.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:13.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:13.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:13.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:14.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:14.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:14.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:14.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:15.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:15.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:15.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:15.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:15.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:34:15.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:15.709638+0000) 2019-10-18T03:34:16.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:16.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:16.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:16.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:16.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:16.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:17.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:17.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:17.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:17.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:18.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:18.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:18.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:18.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:19.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:19.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:19.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:19.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:19.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:19.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:19.356726+0000) 2019-10-18T03:34:20.239 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:34:20.236+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:20.240500+0000) 2019-10-18T03:34:20.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:20.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:20.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:20.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:20.864 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:34:20.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:20.865664+0000) 2019-10-18T03:34:21.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:21.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:21.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:21.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:21.839 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:21.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:22.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:22.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:22.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:22.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:23.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:23.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:23.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:23.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:24.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:24.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:24.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:24.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:25.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:25.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:25.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:25.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:25.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:34:25.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:25.709734+0000) 2019-10-18T03:34:26.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:26.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:26.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:26.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:26.839 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:26.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:27.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:27.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:27.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:27.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:28.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:28.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:29.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:29.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:29.356837+0000) 2019-10-18T03:34:29.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:29.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:29.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:29.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:30.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:34:30.235+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:30.240599+0000) 2019-10-18T03:34:30.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:30.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:30.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:30.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:30.865 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:34:30.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:30.865846+0000) 2019-10-18T03:34:31.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:31.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:31.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:31.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:31.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:31.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:32.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:32.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:32.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:32.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:33.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:33.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:33.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:33.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:34.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:34.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:34.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:34.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:35.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:35.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:35.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:35.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:35.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:34:35.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:35.709830+0000) 2019-10-18T03:34:36.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:36.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:36.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:36.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:36.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:36.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:37.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:37.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:37.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:37.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:38.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:38.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:38.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:38.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:39.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:39.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:39.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:39.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:39.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:39.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:39.356952+0000) 2019-10-18T03:34:40.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:34:40.235+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:40.240702+0000) 2019-10-18T03:34:40.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:40.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:40.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:40.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:40.868 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:34:40.859+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:40.866029+0000) 2019-10-18T03:34:41.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:41.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:41.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:41.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:41.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:41.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:42.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:42.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:42.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:42.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:43.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:43.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:43.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:43.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:44.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:44.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:44.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:44.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:45.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:45.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:45.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:45.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:45.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:34:45.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:45.709925+0000) 2019-10-18T03:34:46.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:46.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:46.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:46.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:46.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:46.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:47.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:47.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:47.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:47.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:48.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:48.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:48.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:48.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:49.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:49.351+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:49.357063+0000) 2019-10-18T03:34:49.356 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:49.351+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:49.357833+0000) 2019-10-18T03:34:49.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:49.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:49.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:49.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:50.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:34:50.235+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:50.240803+0000) 2019-10-18T03:34:50.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:50.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:50.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:50.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:50.865 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:34:50.863+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:50.866240+0000) 2019-10-18T03:34:51.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:51.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:51.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:51.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:51.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:51.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:52.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:52.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:52.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:52.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:53.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:53.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:53.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:53.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:53.856 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:34:53.850+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:53.857291+0000) 2019-10-18T03:34:54.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:54.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:54.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:54.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:55.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:55.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:55.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:55.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:55.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:34:55.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:34:55.710026+0000) 2019-10-18T03:34:56.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:56.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:56.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:56.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:56.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:34:56.838+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:34:57.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:57.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:57.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:57.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:58.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:58.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:58.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:58.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:34:59.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:59.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:34:59.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:34:59.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:00.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:35:00.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:00.240910+0000) 2019-10-18T03:35:00.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:00.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:00.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:00.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:00.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:35:00.862+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:00.866435+0000) 2019-10-18T03:35:01.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:01.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:01.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:01.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:01.845 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:01.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:02.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:02.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:02.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:02.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:03.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:03.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:03.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:03.856 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:35:03.850+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:03.857393+0000) 2019-10-18T03:35:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:04.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:04.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:05.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:05.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:05.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:05.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:05.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:35:05.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:05.710128+0000) 2019-10-18T03:35:06.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:06.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:06.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:06.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:06.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:06.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:07.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:07.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:07.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:07.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:08.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:08.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:08.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:08.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:09.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:09.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:09.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:09.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:10.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:35:10.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:10.241013+0000) 2019-10-18T03:35:10.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:10.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:10.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:10.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:10.865 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:35:10.862+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:10.866626+0000) 2019-10-18T03:35:11.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:11.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:11.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:11.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:11.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:11.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:12.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:12.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:12.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:12.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:13.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:13.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:13.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:13.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:13.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:35:13.850+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:13.857509+0000) 2019-10-18T03:35:14.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:14.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:14.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:14.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:15.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:15.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:15.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:15.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:15.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:35:15.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:15.710228+0000) 2019-10-18T03:35:16.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:16.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:16.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:16.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:16.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:16.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:17.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:17.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:17.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:17.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:18.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:18.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:18.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:18.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:19.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:19.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:19.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:19.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:20.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:35:20.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:20.241115+0000) 2019-10-18T03:35:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:20.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:20.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:20.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:20.866 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:35:20.862+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:20.866811+0000) 2019-10-18T03:35:21.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:21.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:21.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:21.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:21.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:21.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:22.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:22.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:22.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:22.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:23.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:23.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:23.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:23.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:23.856 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:35:23.850+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:23.857609+0000) 2019-10-18T03:35:24.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:24.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:24.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:24.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:25.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:25.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:25.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:25.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:25.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:35:25.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:25.710326+0000) 2019-10-18T03:35:26.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:26.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:26.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:26.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:26.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:26.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:27.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:27.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:27.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:27.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:28.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:28.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:28.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:28.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:29.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:29.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:29.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:29.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:30.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:35:30.234+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:30.241217+0000) 2019-10-18T03:35:30.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:30.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:30.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:30.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:30.866 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:35:30.862+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:30.866996+0000) 2019-10-18T03:35:31.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:31.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:31.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:31.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:31.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:31.845+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:32.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:32.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:33.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:33.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:33.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:33.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:33.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:35:33.850+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:33.857709+0000) 2019-10-18T03:35:34.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:34.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:34.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:34.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:35.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:35.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:35.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:35.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:35.709 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:35:35.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:35.710427+0000) 2019-10-18T03:35:36.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:36.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:36.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:36.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:36.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:36.845+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:37.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:37.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:37.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:37.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:38.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:38.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:38.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:38.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:39.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:39.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:39.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:39.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:40.240 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:35:40.233+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:40.241331+0000) 2019-10-18T03:35:40.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:40.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:40.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:40.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:40.866 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:35:40.862+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:40.867176+0000) 2019-10-18T03:35:41.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:41.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:41.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:41.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:41.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:41.845+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:42.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:42.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:42.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:42.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:43.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:43.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:43.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:43.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:43.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:35:43.850+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:43.857821+0000) 2019-10-18T03:35:44.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:44.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:44.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:44.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:45.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:45.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:45.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:45.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:45.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:35:45.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:45.710527+0000) 2019-10-18T03:35:46.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:46.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:46.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:46.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:46.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:46.845+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:47.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:47.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:47.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:47.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:48.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:48.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:48.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:48.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:49.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:49.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:49.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:49.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:50.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:35:50.233+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:50.241432+0000) 2019-10-18T03:35:50.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:50.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:50.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:50.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:50.866 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:35:50.862+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:50.867320+0000) 2019-10-18T03:35:51.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:51.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:51.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:51.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:51.853 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:51.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:52.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:52.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:52.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:52.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:53.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:53.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:53.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:53.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:53.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:35:53.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:53.857929+0000) 2019-10-18T03:35:54.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:54.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:54.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:54.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:55.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:55.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:55.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:55.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:55.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:35:55.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:35:55.710630+0000) 2019-10-18T03:35:56.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:56.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:56.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:56.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:56.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:35:56.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:35:57.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:57.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:57.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:57.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:58.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:58.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:58.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:58.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:35:59.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:59.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:35:59.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:35:59.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:00.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:36:00.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:00.241554+0000) 2019-10-18T03:36:00.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:00.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:00.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:00.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:00.866 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:36:00.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:00.867536+0000) 2019-10-18T03:36:01.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:01.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:01.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:01.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:01.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:01.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:02.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:02.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:02.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:02.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:03.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:03.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:03.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:03.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:03.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:36:03.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:03.858045+0000) 2019-10-18T03:36:04.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:04.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:04.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:04.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:05.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:05.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:05.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:05.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:05.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:36:05.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:05.710728+0000) 2019-10-18T03:36:06.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:06.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:06.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:06.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:06.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:06.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:07.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:07.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:07.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:07.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:08.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:08.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:08.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:08.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:09.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:09.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:09.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:09.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:10.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:36:10.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:10.241655+0000) 2019-10-18T03:36:10.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:10.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:10.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:10.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:10.867 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:36:10.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:10.867716+0000) 2019-10-18T03:36:11.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:11.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:11.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:11.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:11.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:11.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:12.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:12.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:12.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:12.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:13.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:13.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:13.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:13.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:13.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:36:13.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:13.858159+0000) 2019-10-18T03:36:14.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:14.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:14.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:14.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:15.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:15.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:15.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:15.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:15.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:36:15.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:15.710826+0000) 2019-10-18T03:36:16.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:16.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:16.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:16.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:16.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:16.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:17.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:17.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:17.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:17.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:18.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:18.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:18.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:18.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:19.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:19.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:19.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:19.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:20.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:36:20.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:20.241758+0000) 2019-10-18T03:36:20.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:20.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:20.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:20.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:20.867 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:36:20.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:20.867897+0000) 2019-10-18T03:36:21.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:21.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:21.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:21.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:21.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:21.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:22.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:22.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:22.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:22.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:23.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:23.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:23.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:23.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:23.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:36:23.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:23.858276+0000) 2019-10-18T03:36:24.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:24.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:24.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:24.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:25.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:25.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:25.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:25.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:25.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:36:25.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:25.710923+0000) 2019-10-18T03:36:26.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:26.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:26.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:26.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:26.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:26.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:27.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:27.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:27.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:27.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:28.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:28.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:28.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:28.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:29.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:29.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:29.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:29.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:30.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:30.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:30.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:30.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:30.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:36:30.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:30.241858+0000) 2019-10-18T03:36:30.867 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:36:30.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:30.868079+0000) 2019-10-18T03:36:31.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:31.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:31.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:31.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:31.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:31.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:32.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:32.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:32.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:32.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:33.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:33.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:33.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:33.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:33.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:36:33.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:33.858373+0000) 2019-10-18T03:36:34.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:34.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:34.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:34.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:35.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:35.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:35.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:35.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:35.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:36:35.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:35.711035+0000) 2019-10-18T03:36:36.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:36.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:36.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:36.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:36.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:36.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:37.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:37.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:37.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:37.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:38.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:38.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:38.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:38.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:39.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:39.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:39.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:39.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:40.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:40.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:40.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:40.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:40.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:36:40.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:40.241958+0000) 2019-10-18T03:36:40.867 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:36:40.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:40.868284+0000) 2019-10-18T03:36:41.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:41.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:41.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:41.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:41.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:41.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:42.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:42.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:42.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:42.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:43.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:43.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:43.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:43.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:43.857 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:36:43.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:43.858473+0000) 2019-10-18T03:36:44.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:44.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:44.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:44.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:45.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:45.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:45.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:45.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:36:45.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:45.711135+0000) 2019-10-18T03:36:45.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:45.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:45.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:45.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:46.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:46.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:46.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:46.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:46.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:46.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:47.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:47.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:47.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:47.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:48.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:48.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:48.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:48.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:49.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:49.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:49.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:49.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:50.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:36:50.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:50.242063+0000) 2019-10-18T03:36:50.867 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:36:50.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:50.868472+0000) 2019-10-18T03:36:50.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:50.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:50.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:50.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:51.863 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:51.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:51.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:51.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:51.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:51.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:52.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:52.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:52.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:52.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:53.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:36:53.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:53.858570+0000) 2019-10-18T03:36:53.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:53.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:53.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:53.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:54.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:54.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:54.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:54.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:55.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:36:55.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:36:55.711237+0000) 2019-10-18T03:36:55.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:55.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:55.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:55.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:56.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:36:56.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:36:56.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:56.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:56.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:56.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:57.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:57.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:57.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:57.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:58.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:58.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:58.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:58.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:36:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:59.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:36:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:36:59.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:00.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:37:00.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:00.242170+0000) 2019-10-18T03:37:00.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:00.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:00.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:00.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:00.867 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:37:00.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:00.868664+0000) 2019-10-18T03:37:01.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:01.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:01.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:01.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:01.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:01.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:02.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:02.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:02.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:02.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:03.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:37:03.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:03.858680+0000) 2019-10-18T03:37:03.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:03.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:03.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:03.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:04.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:04.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:04.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:04.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:05.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:37:05.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:05.711341+0000) 2019-10-18T03:37:05.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:05.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:05.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:05.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:06.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:06.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:06.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:06.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:06.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:06.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:07.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:07.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:07.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:07.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:08.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:08.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:08.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:08.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:09.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:09.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:09.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:09.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:10.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:37:10.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:10.242271+0000) 2019-10-18T03:37:10.868 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:37:10.865+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:10.868846+0000) 2019-10-18T03:37:10.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:10.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:10.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:10.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:11.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:11.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:11.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:11.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:11.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:11.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:12.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:12.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:12.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:12.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:13.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:37:13.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:13.858791+0000) 2019-10-18T03:37:13.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:13.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:13.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:13.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:14.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:14.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:14.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:14.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:15.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:37:15.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:15.711443+0000) 2019-10-18T03:37:15.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:15.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:15.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:15.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:16.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:16.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:16.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:16.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:16.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:16.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:18.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:18.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:18.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:18.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:19.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:19.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:19.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:19.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:20.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:20.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:20.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:20.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:20.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:37:20.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:20.242373+0000) 2019-10-18T03:37:20.868 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:37:20.864+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:20.869064+0000) 2019-10-18T03:37:21.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:21.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:21.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:21.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:21.868 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:21.862+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:22.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:22.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:22.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:22.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:23.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:23.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:23.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:23.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:23.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:37:23.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:23.858900+0000) 2019-10-18T03:37:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:24.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:24.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:25.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:25.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:25.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:25.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:25.710 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:37:25.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:25.711542+0000) 2019-10-18T03:37:26.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:26.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:26.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:26.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:26.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:26.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:27.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:27.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:27.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:27.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:28.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:28.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:28.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:28.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:29.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:29.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:29.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:29.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:30.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:30.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:30.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:30.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:30.241 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:37:30.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:30.242473+0000) 2019-10-18T03:37:30.868 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:37:30.864+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:30.869271+0000) 2019-10-18T03:37:31.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:31.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:31.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:31.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:31.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:31.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:32.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:32.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:32.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:32.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:33.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:33.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:33.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:33.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:33.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:37:33.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:33.859018+0000) 2019-10-18T03:37:34.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:34.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:34.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:34.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:35.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:35.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:35.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:35.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:35.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:37:35.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:35.711646+0000) 2019-10-18T03:37:36.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:36.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:36.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:36.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:36.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:36.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:37.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:37.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:37.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:37.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:38.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:38.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:38.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:38.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:39.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:39.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:39.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:39.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:40.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:40.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:40.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:40.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:40.242 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:37:40.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:40.242574+0000) 2019-10-18T03:37:40.868 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:37:40.864+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:40.869452+0000) 2019-10-18T03:37:41.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:41.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:41.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:41.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:41.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:41.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:41.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:41.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:42.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:41.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:43.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:43.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:43.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:43.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:43.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:37:43.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:43.859127+0000) 2019-10-18T03:37:43.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:43.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:43.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:43.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:45.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:44.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:45.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:44.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:45.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:37:45.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:45.711750+0000) 2019-10-18T03:37:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:45.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:45.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:46.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:46.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:47.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:46.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:46.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:48.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:48.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:48.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:48.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:48.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:48.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:48.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:48.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:49.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:49.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:49.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:49.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:50.242 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:37:50.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:50.242679+0000) 2019-10-18T03:37:50.868 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:37:50.864+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:50.869634+0000) 2019-10-18T03:37:51.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:51.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:51.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:51.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:51.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:51.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:52.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:52.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:52.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:52.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:53.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:53.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:53.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:53.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:53.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:37:53.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:53.859237+0000) 2019-10-18T03:37:54.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:54.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:54.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:54.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:55.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:55.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:55.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:55.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:55.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:37:55.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:37:55.711850+0000) 2019-10-18T03:37:56.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:56.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:56.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:56.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:56.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:37:56.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:37:57.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:57.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:57.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:57.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:58.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:58.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:58.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:58.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:37:59.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:59.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:37:59.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:37:59.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:00.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:00.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:00.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:00.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:00.242 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:38:00.237+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:00.242786+0000) 2019-10-18T03:38:00.869 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:38:00.864+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:00.869848+0000) 2019-10-18T03:38:01.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:01.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:01.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:01.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:01.875 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:01.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:02.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:02.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:02.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:02.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:03.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:03.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:03.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:03.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:03.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:38:03.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:03.859346+0000) 2019-10-18T03:38:04.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:04.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:04.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:04.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:05.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:05.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:05.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:05.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:05.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:38:05.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:05.711953+0000) 2019-10-18T03:38:06.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:06.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:06.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:06.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:06.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:06.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:07.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:07.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:07.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:07.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:08.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:08.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:08.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:08.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:09.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:09.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:09.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:09.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:10.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:10.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:10.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:10.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:10.242 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:38:10.237+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:10.242886+0000) 2019-10-18T03:38:10.869 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:38:10.864+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:10.870029+0000) 2019-10-18T03:38:11.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:11.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:11.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:11.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:11.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:11.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:12.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:12.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:12.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:12.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:13.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:13.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:13.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:13.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:13.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:38:13.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:13.859459+0000) 2019-10-18T03:38:14.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:14.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:14.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:14.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:15.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:15.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:15.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:15.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:15.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:38:15.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:15.712051+0000) 2019-10-18T03:38:16.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:16.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:16.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:16.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:16.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:16.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:17.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:17.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:17.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:17.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:18.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:18.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:18.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:18.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:19.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:19.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:19.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:19.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:20.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:20.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:20.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:20.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:20.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:38:20.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:20.242990+0000) 2019-10-18T03:38:20.869 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:38:20.868+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:20.870175+0000) 2019-10-18T03:38:20.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:20.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:20.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:20.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:21.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:21.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:21.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:21.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:21.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:21.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:22.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:22.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:22.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:22.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:23.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:38:23.856+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:23.859571+0000) 2019-10-18T03:38:23.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:23.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:23.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:23.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:24.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:24.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:24.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:24.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:25.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:38:25.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:25.712155+0000) 2019-10-18T03:38:25.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:25.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:25.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:25.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:26.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:26.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:26.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:26.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:26.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:26.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:27.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:27.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:27.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:27.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:28.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:28.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:28.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:28.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:29.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:29.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:29.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:29.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:30.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:38:30.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:30.243090+0000) 2019-10-18T03:38:30.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:30.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:30.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:30.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:30.869 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:38:30.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:30.870375+0000) 2019-10-18T03:38:31.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:31.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:31.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:31.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:31.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:31.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:32.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:32.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:32.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:32.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:33.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:33.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:33.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:33.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:33.858 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:38:33.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:33.859684+0000) 2019-10-18T03:38:34.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:34.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:34.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:34.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:35.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:38:35.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:35.712255+0000) 2019-10-18T03:38:35.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:35.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:35.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:35.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:36.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:36.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:36.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:36.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:36.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:36.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:37.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:37.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:37.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:37.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:38.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:38.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:38.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:38.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:39.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:39.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:39.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:39.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:40.242 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:38:40.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:40.243190+0000) 2019-10-18T03:38:40.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:40.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:40.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:40.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:40.869 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:38:40.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:40.870560+0000) 2019-10-18T03:38:41.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:41.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:41.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:41.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:41.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:41.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:42.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:42.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:42.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:42.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:43.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:43.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:43.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:43.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:43.859 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:38:43.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:43.859799+0000) 2019-10-18T03:38:44.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:44.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:44.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:44.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:45.711 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:38:45.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:45.712355+0000) 2019-10-18T03:38:45.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:45.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:45.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:45.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:46.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:46.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:46.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:46.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:46.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:46.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:47.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:47.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:47.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:47.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:48.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:48.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:48.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:48.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:49.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:49.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:49.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:49.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:50.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:38:50.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:50.243295+0000) 2019-10-18T03:38:50.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:50.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:50.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:50.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:50.870 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:38:50.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:50.870741+0000) 2019-10-18T03:38:51.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:51.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:51.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:51.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:51.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:51.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:52.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:52.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:52.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:52.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:53.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:53.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:53.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:53.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:53.859 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:38:53.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:53.859916+0000) 2019-10-18T03:38:54.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:54.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:54.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:54.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:55.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:38:55.708+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:38:55.712455+0000) 2019-10-18T03:38:55.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:55.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:55.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:55.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:56.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:56.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:56.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:56.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:56.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:38:56.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:38:57.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:57.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:57.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:57.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:58.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:58.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:58.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:58.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:38:59.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:59.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:38:59.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:38:59.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:00.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:39:00.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:00.243403+0000) 2019-10-18T03:39:00.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:00.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:00.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:00.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:00.870 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:39:00.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:00.870939+0000) 2019-10-18T03:39:01.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:01.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:01.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:01.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:01.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:01.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:02.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:02.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:02.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:02.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:03.859 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:39:03.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:03.860028+0000) 2019-10-18T03:39:03.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:03.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:03.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:03.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:04.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:04.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:04.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:04.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:05.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:39:05.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:05.712553+0000) 2019-10-18T03:39:05.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:05.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:05.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:05.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:06.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:06.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:06.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:06.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:06.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:06.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:07.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:07.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:08.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:08.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:08.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:08.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:09.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:09.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:09.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:09.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:10.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:39:10.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:10.243504+0000) 2019-10-18T03:39:10.873 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:39:10.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:10.871121+0000) 2019-10-18T03:39:10.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:10.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:10.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:10.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:11.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:11.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:11.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:11.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:12.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:12.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:12.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:12.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:13.859 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:39:13.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:13.860143+0000) 2019-10-18T03:39:13.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:13.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:13.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:13.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:14.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:14.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:15.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:39:15.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:15.712650+0000) 2019-10-18T03:39:15.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:15.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:15.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:15.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:16.887 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:16.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:16.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:16.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:16.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:16.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:17.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:17.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:17.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:17.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:18.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:18.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:18.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:18.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:19.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:19.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:19.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:19.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:20.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:39:20.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:20.243608+0000) 2019-10-18T03:39:20.870 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:39:20.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:20.871306+0000) 2019-10-18T03:39:20.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:20.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:20.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:20.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:21.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:21.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:21.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:21.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:21.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:21.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:23.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:23.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:23.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:23.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:23.859 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:39:23.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:23.860253+0000) 2019-10-18T03:39:24.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:24.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:24.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:24.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:25.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:24.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:25.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:24.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:25.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:39:25.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:25.712750+0000) 2019-10-18T03:39:25.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:25.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:25.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:25.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:26.889 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:26.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:27.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:27.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:27.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:27.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:28.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:28.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:28.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:28.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:29.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:29.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:29.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:29.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:30.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:30.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:30.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:30.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:30.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:39:30.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:30.243706+0000) 2019-10-18T03:39:30.870 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:39:30.867+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:30.871487+0000) 2019-10-18T03:39:31.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:30.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:31.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:30.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:31.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:31.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:31.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:31.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:31.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:31.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:32.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:32.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:32.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:32.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:33.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:39:33.855+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:33.860365+0000) 2019-10-18T03:39:33.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:33.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:33.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:33.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:34.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:34.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:34.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:34.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:35.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:39:35.707+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:35.712852+0000) 2019-10-18T03:39:35.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:35.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:35.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:35.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:36.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:36.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:36.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:36.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:36.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:36.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:37.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:37.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:37.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:37.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:38.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:38.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:38.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:38.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:39.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:39.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:39.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:39.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:40.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:39:40.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:40.243806+0000) 2019-10-18T03:39:40.870 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:39:40.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:40.871668+0000) 2019-10-18T03:39:40.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:40.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:40.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:40.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:41.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:41.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:41.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:41.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:41.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:41.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:42.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:42.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:42.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:42.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:43.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:39:43.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:43.860476+0000) 2019-10-18T03:39:44.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:44.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:44.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:44.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:45.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:45.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:45.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:45.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:45.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:39:45.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:45.712951+0000) 2019-10-18T03:39:46.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:46.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:46.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:46.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:46.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:46.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:47.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:47.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:47.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:47.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:47.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:47.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:47.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:47.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:48.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:48.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:48.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:48.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:49.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:49.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:49.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:49.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:50.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:39:50.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:50.243905+0000) 2019-10-18T03:39:50.871 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:39:50.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:50.871855+0000) 2019-10-18T03:39:50.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:50.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:50.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:50.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:51.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:51.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:51.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:51.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:51.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:51.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:52.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:52.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:52.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:52.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:53.859 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:39:53.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:53.860584+0000) 2019-10-18T03:39:53.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:53.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:53.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:53.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:54.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:54.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:54.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:54.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:55.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:39:55.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:39:55.713049+0000) 2019-10-18T03:39:55.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:55.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:55.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:55.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:56.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:39:56.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:39:56.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:56.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:56.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:56.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:57.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:57.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:57.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:57.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:58.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:58.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:58.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:58.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:39:59.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:59.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:39:59.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:39:59.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:00.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:40:00.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:00.244011+0000) 2019-10-18T03:40:00.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:00.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:00.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:00.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:00.871 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:40:00.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:00.872042+0000) 2019-10-18T03:40:01.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:01.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:01.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:01.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:01.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:01.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:02.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:02.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:02.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:02.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 360 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:03.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:03.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:03.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:03.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:03.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:40:03.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:03.860698+0000) 2019-10-18T03:40:04.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:04.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:04.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:04.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:05.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:40:05.706+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:05.713148+0000) 2019-10-18T03:40:05.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:05.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:05.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:05.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:06.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:06.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:06.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:06.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:06.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:06.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:07.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:07.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:07.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:07.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:08.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:08.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:08.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:08.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:09.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:09.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:10.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:40:10.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:10.244113+0000) 2019-10-18T03:40:10.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:10.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:10.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:10.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:10.872 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:40:10.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:10.872232+0000) 2019-10-18T03:40:11.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:11.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:11.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:11.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:11.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:11.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:12.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:12.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:12.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:12.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:13.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:13.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:13.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:13.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:13.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:40:13.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:13.860813+0000) 2019-10-18T03:40:14.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:14.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:14.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:14.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:15.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:40:15.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:15.713246+0000) 2019-10-18T03:40:15.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:15.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:15.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:15.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:16.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:16.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:16.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:16.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:16.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:17.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:17.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:17.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:17.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:18.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:18.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:18.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:18.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:19.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:19.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:19.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:19.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:20.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:40:20.237+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:20.244216+0000) 2019-10-18T03:40:20.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:20.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:20.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:20.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:20.871 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:40:20.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:20.872452+0000) 2019-10-18T03:40:21.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:21.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:21.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:21.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:21.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:21.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:22.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:22.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:22.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:22.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:23.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:23.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:23.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:23.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:23.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:40:23.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:23.860921+0000) 2019-10-18T03:40:24.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:24.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:24.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:24.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:25.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:25.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:25.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:25.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:25.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:40:25.705+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:25.713348+0000) 2019-10-18T03:40:26.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:26.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:26.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:26.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:26.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:26.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:27.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:27.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:27.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:27.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:28.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:28.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:28.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:28.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:29.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:29.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:29.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:29.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:30.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:40:30.237+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:30.244316+0000) 2019-10-18T03:40:30.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:30.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:30.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:30.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:30.871 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:40:30.866+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:30.872666+0000) 2019-10-18T03:40:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:31.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:31.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:31.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:31.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:31.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:32.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:32.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:32.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:32.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:33.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:33.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:33.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:33.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:33.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:40:33.854+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:33.861032+0000) 2019-10-18T03:40:34.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:34.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:34.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:34.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:35.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:35.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:35.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:35.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:35.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:40:35.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:35.713455+0000) 2019-10-18T03:40:36.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:36.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:36.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:36.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:36.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:36.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:37.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:37.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:37.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:37.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:38.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:38.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:38.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:38.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:39.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:39.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:39.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:39.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:40.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:40:40.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:40.244417+0000) 2019-10-18T03:40:40.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:40.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:40.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:40.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:40.872 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:40:40.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:40.872806+0000) 2019-10-18T03:40:41.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:41.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:41.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:41.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:41.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:41.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:42.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:42.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:42.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:42.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:43.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:43.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:43.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:43.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:43.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:40:43.858+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:43.861144+0000) 2019-10-18T03:40:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:44.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:44.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:45.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:45.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:45.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:45.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:45.712 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:40:45.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:45.713553+0000) 2019-10-18T03:40:46.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:46.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:46.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:46.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:46.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:46.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:47.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:47.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:47.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:47.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:48.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:48.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:48.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:48.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:49.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:49.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:49.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:49.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:50.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:40:50.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:50.244523+0000) 2019-10-18T03:40:50.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:50.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:50.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:50.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:50.872 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:40:50.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:50.872995+0000) 2019-10-18T03:40:51.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:51.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:51.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:51.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:51.904 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:51.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:52.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:52.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:52.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:52.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:53.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:53.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:53.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:53.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:53.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:40:53.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:53.861257+0000) 2019-10-18T03:40:54.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:54.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:54.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:54.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:55.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:55.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:55.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:55.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:55.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:40:55.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:40:55.713661+0000) 2019-10-18T03:40:56.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:56.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:56.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:56.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:56.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:40:56.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:40:57.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:57.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:57.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:57.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:58.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:58.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:58.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:58.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:40:59.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:59.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:40:59.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:40:59.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:00.243 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:41:00.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:00.244630+0000) 2019-10-18T03:41:00.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:00.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:00.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:00.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:00.872 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:41:00.869+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:00.873154+0000) 2019-10-18T03:41:01.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:01.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:01.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:01.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:01.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:01.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:02.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:02.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:02.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:02.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:03.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:03.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:03.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:03.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:03.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:41:03.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:03.861369+0000) 2019-10-18T03:41:04.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:04.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:04.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:04.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:05.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:05.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:05.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:05.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:05.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:41:05.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:05.713764+0000) 2019-10-18T03:41:06.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:06.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:06.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:06.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:06.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:06.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:07.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:07.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:07.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:07.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:08.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:08.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:08.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:08.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:09.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:09.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:09.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:09.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:10.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:41:10.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:10.244732+0000) 2019-10-18T03:41:10.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:10.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:10.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:10.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:10.872 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:41:10.869+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:10.873366+0000) 2019-10-18T03:41:11.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:11.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:11.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:11.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:11.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:11.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:12.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:12.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:12.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:12.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:13.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:13.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:13.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:13.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:13.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:41:13.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:13.861483+0000) 2019-10-18T03:41:14.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:14.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:14.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:14.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:15.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:15.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:15.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:15.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:15.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:41:15.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:15.713882+0000) 2019-10-18T03:41:16.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:16.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:16.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:16.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:16.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:16.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:17.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:17.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:17.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:17.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:18.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:18.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:18.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:18.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:19.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:19.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:19.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:19.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:20.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:41:20.240+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:20.244840+0000) 2019-10-18T03:41:20.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:20.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:20.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:20.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:20.872 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:41:20.869+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:20.873572+0000) 2019-10-18T03:41:21.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:21.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:21.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:21.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:21.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:21.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:22.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:22.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:22.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:22.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:23.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:23.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:23.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:23.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:23.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:41:23.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:23.861596+0000) 2019-10-18T03:41:24.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:24.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:24.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:24.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:25.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:25.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:25.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:25.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:25.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:41:25.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:25.713981+0000) 2019-10-18T03:41:26.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:26.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:26.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:26.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:26.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:26.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:27.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:27.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:27.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:27.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:28.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:28.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:28.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:28.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:29.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:29.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:29.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:29.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:30.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:41:30.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:30.244939+0000) 2019-10-18T03:41:30.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:30.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:30.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:30.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:30.873 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:41:30.869+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:30.873786+0000) 2019-10-18T03:41:31.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:31.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:31.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:31.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:31.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:31.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:32.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:32.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:32.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:32.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:33.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:33.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:33.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:33.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:33.861 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:41:33.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:33.861763+0000) 2019-10-18T03:41:34.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:34.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:34.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:34.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:35.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:35.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:35.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:35.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:35.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:41:35.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:35.714082+0000) 2019-10-18T03:41:36.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:36.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:36.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:36.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:36.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:36.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:37.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:37.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:37.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:37.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:38.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:38.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:39.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:39.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:39.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:39.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:40.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:41:40.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:40.245040+0000) 2019-10-18T03:41:40.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:40.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:40.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:40.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:40.873 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:41:40.869+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:40.873999+0000) 2019-10-18T03:41:41.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:41.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:41.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:41.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:41.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:41.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:42.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:42.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:42.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:42.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:43.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:43.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:43.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:43.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:43.861 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:41:43.857+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:43.861951+0000) 2019-10-18T03:41:44.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:44.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:44.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:44.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:45.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:45.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:45.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:45.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:45.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:41:45.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:45.714192+0000) 2019-10-18T03:41:46.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:46.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:46.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:46.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:46.913 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:46.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:47.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:47.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:47.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:47.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:48.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:48.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:48.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:48.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:49.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:49.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:49.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:49.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:50.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:41:50.239+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:50.245140+0000) 2019-10-18T03:41:50.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:50.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:50.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:50.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:50.873 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:41:50.873+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:50.874217+0000) 2019-10-18T03:41:51.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:51.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:51.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:51.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:51.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:51.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:52.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:52.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:52.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:52.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:53.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:53.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:53.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:53.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:53.861 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:41:53.861+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:53.862159+0000) 2019-10-18T03:41:54.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:54.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:54.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:54.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:55.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:55.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:55.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:55.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:55.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:41:55.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:41:55.714292+0000) 2019-10-18T03:41:56.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:56.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:56.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:56.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:56.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:41:56.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:41:57.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:57.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:57.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:57.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:58.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:58.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:58.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:58.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:41:59.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:59.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:41:59.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:41:59.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:00.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:42:00.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:00.245244+0000) 2019-10-18T03:42:00.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:00.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:00.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:00.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:00.873 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:42:00.873+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:00.874398+0000) 2019-10-18T03:42:01.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:01.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:01.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:01.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:01.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:01.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:02.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:02.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:02.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:02.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:03.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:03.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:03.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:03.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:03.861 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:42:03.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:03.862360+0000) 2019-10-18T03:42:04.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:04.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:04.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:04.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:05.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:05.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:05.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:05.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:05.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:42:05.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:05.714392+0000) 2019-10-18T03:42:06.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:06.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:06.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:06.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:06.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:06.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:07.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:07.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:07.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:07.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:08.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:08.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:08.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:08.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:09.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:09.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:09.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:09.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:10.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:42:10.238+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:10.245345+0000) 2019-10-18T03:42:10.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:10.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:10.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:10.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:10.873 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:42:10.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:10.874612+0000) 2019-10-18T03:42:11.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:11.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:11.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:11.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:11.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:11.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:12.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:12.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:12.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:12.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:13.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:13.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:13.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:13.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:13.861 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:42:13.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:13.862540+0000) 2019-10-18T03:42:14.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:14.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:14.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:14.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:15.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:15.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:15.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:15.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:15.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:42:15.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:15.714491+0000) 2019-10-18T03:42:16.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:16.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:16.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:16.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:16.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:16.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:17.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:17.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:17.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:17.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:18.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:18.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:18.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:18.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:19.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:19.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:19.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:19.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:20.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:42:20.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:20.245454+0000) 2019-10-18T03:42:20.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:20.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:20.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:20.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:20.874 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:42:20.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:20.874814+0000) 2019-10-18T03:42:21.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:21.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:21.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:21.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:21.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:21.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:22.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:22.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:22.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:22.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:23.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:23.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:23.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:23.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:23.862 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:42:23.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:23.862729+0000) 2019-10-18T03:42:24.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:24.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:24.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:24.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:25.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:25.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:25.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:25.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:25.713 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:42:25.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:25.714593+0000) 2019-10-18T03:42:26.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:26.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:26.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:26.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:26.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:26.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:27.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:27.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:27.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:27.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:28.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:28.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:28.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:28.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:29.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:29.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:29.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:29.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:30.244 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:42:30.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:30.245552+0000) 2019-10-18T03:42:30.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:30.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:30.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:30.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:30.874 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:42:30.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:30.874965+0000) 2019-10-18T03:42:31.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:31.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:31.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:31.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:31.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:31.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:32.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:32.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:32.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:32.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:33.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:33.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:33.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:33.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:33.862 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:42:33.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:33.862913+0000) 2019-10-18T03:42:34.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:34.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:34.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:34.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:35.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:35.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:35.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:35.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:35.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:42:35.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:35.714694+0000) 2019-10-18T03:42:36.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:36.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:36.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:36.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:36.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:36.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:37.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:37.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:37.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:37.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:38.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:38.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:38.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:38.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:39.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:39.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:39.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:39.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:40.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:42:40.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:40.245653+0000) 2019-10-18T03:42:40.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:40.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:40.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:40.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:40.874 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:42:40.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:40.875111+0000) 2019-10-18T03:42:41.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:41.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:41.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:41.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:41.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:41.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:42.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:42.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:42.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:42.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:43.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:43.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:43.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:43.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:43.862 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:42:43.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:43.863126+0000) 2019-10-18T03:42:44.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:44.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:44.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:44.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:45.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:45.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:45.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:45.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:45.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:42:45.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:45.714793+0000) 2019-10-18T03:42:46.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:46.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:46.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:46.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:46.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:46.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:47.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:47.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:47.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:47.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:48.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:48.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:48.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:48.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:49.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:49.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:49.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:49.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:50.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:42:50.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:50.245755+0000) 2019-10-18T03:42:50.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:50.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:50.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:50.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:50.874 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:42:50.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:50.875334+0000) 2019-10-18T03:42:51.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:51.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:51.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:51.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:51.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:51.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:52.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:52.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:32:20.214414+0000 (oldest deadline 2019-10-18T03:32:40.214414+0000) 2019-10-18T03:42:52.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:52.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:53.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:53.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:53.862 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:42:53.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:53.863307+0000) 2019-10-18T03:42:54.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:54.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:55.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:55.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:55.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:42:55.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:42:55.714894+0000) 2019-10-18T03:42:56.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:56.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:56.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:42:56.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:42:57.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:57.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:58.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:58.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:42:59.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:42:59.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:00.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:43:00.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:00.245858+0000) 2019-10-18T03:43:00.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:00.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:00.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:43:00.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:00.875498+0000) 2019-10-18T03:43:01.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:01.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:01.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:01.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:02.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:02.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:03.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:03.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:03.862 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:43:03.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:03.863493+0000) 2019-10-18T03:43:04.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:04.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:05.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:05.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:05.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:43:05.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:05.714993+0000) 2019-10-18T03:43:06.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:06.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:06.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:06.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:07.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:07.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:08.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:08.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:09.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:09.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:10.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:43:10.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:10.245953+0000) 2019-10-18T03:43:10.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:10.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:10.874 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:43:10.872+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:10.875683+0000) 2019-10-18T03:43:11.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:11.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:11.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:11.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:12.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:12.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:13.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:13.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:13.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:43:13.860+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:13.863635+0000) 2019-10-18T03:43:14.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:14.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:15.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:15.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:15.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:15.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:15.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:43:15.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:15.715095+0000) 2019-10-18T03:43:16.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:16.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:16.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:16.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:16.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:16.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:17.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:17.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:17.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:17.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:18.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:18.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:18.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:18.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:19.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:19.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:19.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:19.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:20.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:43:20.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:20.246055+0000) 2019-10-18T03:43:20.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:20.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:20.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:20.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:20.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:43:20.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:20.875863+0000) 2019-10-18T03:43:21.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:21.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:21.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:21.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:21.929 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:21.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:22.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:22.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:22.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:22.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:23.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:23.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:23.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:23.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:23.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:43:23.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:23.863776+0000) 2019-10-18T03:43:24.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:24.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:24.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:24.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:25.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:25.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:25.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:25.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:25.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:43:25.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:25.715190+0000) 2019-10-18T03:43:26.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:26.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:26.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:26.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:26.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:26.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:27.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:27.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:27.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:27.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:28.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:28.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:28.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:28.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:29.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:29.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:29.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:29.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:30.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:43:30.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:30.246152+0000) 2019-10-18T03:43:30.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:30.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:30.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:30.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:30.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:43:30.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:30.876045+0000) 2019-10-18T03:43:31.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:31.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:31.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:31.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:31.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:31.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:32.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:32.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:32.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:33.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:33.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:33.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:33.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:33.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:43:33.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:33.863972+0000) 2019-10-18T03:43:34.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:34.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:34.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:34.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:35.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:35.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:35.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:43:35.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:35.715292+0000) 2019-10-18T03:43:36.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:36.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:36.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:36.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:36.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:36.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:37.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:37.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:37.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:37.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:38.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:38.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:38.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:38.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:39.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:39.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:39.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:39.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:40.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:43:40.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:40.246253+0000) 2019-10-18T03:43:40.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:40.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:40.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:40.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:40.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:43:40.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:40.876238+0000) 2019-10-18T03:43:41.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:41.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:41.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:41.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:41.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:41.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:42.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:42.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:42.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:42.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:43.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:43.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:43.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:43.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:43.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:43:43.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:43.864156+0000) 2019-10-18T03:43:44.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:44.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:44.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:44.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:45.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:45.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:45.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:45.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:45.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:43:45.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:45.715396+0000) 2019-10-18T03:43:46.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:46.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:46.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:46.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:46.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:46.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:47.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:47.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:47.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:47.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:48.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:48.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:48.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:48.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:49.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:49.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:49.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:49.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:50.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:43:50.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:50.246357+0000) 2019-10-18T03:43:50.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:50.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:50.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:50.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:50.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:43:50.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:50.876455+0000) 2019-10-18T03:43:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:51.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:51.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:51.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:51.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:52.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:52.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:52.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:52.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:53.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:53.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:53.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:43:53.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:53.864363+0000) 2019-10-18T03:43:54.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:54.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:54.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:54.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:55.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:55.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:55.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:55.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:43:55.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:43:55.715500+0000) 2019-10-18T03:43:56.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:56.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:56.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:56.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:56.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:43:56.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:43:57.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:57.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:57.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:57.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:58.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:58.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:58.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:58.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:43:59.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:59.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:43:59.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:43:59.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:00.245 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:44:00.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:00.246460+0000) 2019-10-18T03:44:00.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:00.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:00.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:00.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:00.875 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:44:00.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:00.876674+0000) 2019-10-18T03:44:01.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:01.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:01.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:01.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:01.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:01.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:02.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:02.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:02.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:02.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:03.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:03.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:03.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:03.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:03.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:44:03.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:03.864574+0000) 2019-10-18T03:44:04.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:04.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:04.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:04.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:05.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:05.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:05.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:05.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:05.714 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:44:05.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:05.715602+0000) 2019-10-18T03:44:06.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:06.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:06.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:06.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:06.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:06.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:07.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:07.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:07.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:07.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:08.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:08.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:08.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:08.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:09.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:09.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:09.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:09.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:10.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:44:10.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:10.246563+0000) 2019-10-18T03:44:10.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:10.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:10.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:10.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:10.876 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:44:10.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:10.876855+0000) 2019-10-18T03:44:11.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:11.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:11.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:11.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:11.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:11.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:12.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:12.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:12.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:12.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:13.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:13.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:13.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:13.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:13.863 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:44:13.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:13.864762+0000) 2019-10-18T03:44:14.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:14.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:14.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:14.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:15.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:15.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:15.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:15.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:15.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:44:15.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:15.715704+0000) 2019-10-18T03:44:16.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:16.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:16.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:16.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:16.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:16.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:17.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:17.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:17.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:17.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:18.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:18.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:18.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:18.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:19.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:19.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:20.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:44:20.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:20.246666+0000) 2019-10-18T03:44:20.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:20.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:20.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:20.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:20.876 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:44:20.871+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:20.877044+0000) 2019-10-18T03:44:21.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:21.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:21.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:21.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:21.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:21.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:22.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:22.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:22.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:22.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:23.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:23.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:23.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:23.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:23.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:44:23.859+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:23.864948+0000) 2019-10-18T03:44:24.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:24.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:24.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:24.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:25.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:25.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:25.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:25.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:25.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:44:25.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:25.715799+0000) 2019-10-18T03:44:26.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:26.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:26.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:26.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:26.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:26.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:27.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:27.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:27.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:27.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:28.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:28.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:28.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:28.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:29.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:29.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:29.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:29.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:30.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:44:30.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:30.246765+0000) 2019-10-18T03:44:30.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:30.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:30.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:30.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:30.876 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:44:30.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:30.877231+0000) 2019-10-18T03:44:31.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:31.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:31.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:31.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:31.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:31.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:32.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:32.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:32.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:32.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:33.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:33.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:33.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:33.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:33.864 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:44:33.858+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:33.865157+0000) 2019-10-18T03:44:34.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:34.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:34.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:34.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:35.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:35.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:35.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:35.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:35.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:44:35.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:35.715901+0000) 2019-10-18T03:44:36.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:36.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:36.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:36.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:36.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:36.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:37.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:37.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:37.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:37.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:38.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:38.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:38.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:38.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:39.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:39.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:39.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:39.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:40.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:44:40.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:40.246862+0000) 2019-10-18T03:44:40.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:40.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:40.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:40.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:40.876 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:44:40.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:40.877421+0000) 2019-10-18T03:44:41.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:41.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:41.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:41.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:41.942 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:41.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:42.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:42.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:42.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:42.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:43.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:43.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:43.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:43.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:43.864 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:44:43.858+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:43.865338+0000) 2019-10-18T03:44:44.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:44.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:44.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:44.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:45.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:45.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:45.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:45.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:44:45.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:45.715997+0000) 2019-10-18T03:44:46.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:46.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:46.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:46.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:46.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:46.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:47.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:47.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:47.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:47.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:48.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:48.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:48.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:48.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:49.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:49.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:49.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:49.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:50.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:44:50.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:50.246967+0000) 2019-10-18T03:44:50.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:50.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:50.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:50.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:50.876 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:44:50.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:50.877609+0000) 2019-10-18T03:44:51.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:51.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:51.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:51.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:51.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:51.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:52.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:52.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:52.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:52.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:53.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:53.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:53.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:53.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:53.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:44:53.858+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:53.865528+0000) 2019-10-18T03:44:54.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:54.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:54.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:54.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:55.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:55.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:55.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:55.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:55.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:44:55.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:44:55.716102+0000) 2019-10-18T03:44:56.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:56.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:56.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:56.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:56.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:44:56.937+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:44:57.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:57.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:57.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:57.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:58.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:58.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:58.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:44:59.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:59.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:44:59.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:44:59.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:00.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:45:00.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:00.247069+0000) 2019-10-18T03:45:00.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:00.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:00.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:00.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:00.877 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:45:00.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:00.877787+0000) 2019-10-18T03:45:01.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:01.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:01.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:01.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:01.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:01.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:02.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:02.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:02.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:02.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:03.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:03.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:03.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:03.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:03.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:45:03.858+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:03.865739+0000) 2019-10-18T03:45:04.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:04.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:04.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:04.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:05.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:05.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:05.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:05.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:05.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:45:05.709+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:05.716203+0000) 2019-10-18T03:45:06.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:06.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:06.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:06.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:06.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:06.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:07.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:07.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:07.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:07.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:08.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:08.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:08.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:08.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:09.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:09.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:09.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:09.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:10.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:45:10.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:10.247170+0000) 2019-10-18T03:45:10.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:10.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:10.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:10.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:10.877 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:45:10.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:10.877925+0000) 2019-10-18T03:45:11.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:11.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:11.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:11.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:11.947 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:11.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:12.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:12.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:12.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:12.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:13.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:13.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:13.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:13.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:13.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:45:13.858+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:13.865950+0000) 2019-10-18T03:45:14.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:14.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:14.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:14.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:15.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:15.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:15.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:15.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:15.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:45:15.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:15.716306+0000) 2019-10-18T03:45:16.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:16.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:16.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:16.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:16.948 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:16.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:17.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:17.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:17.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:17.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:18.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:18.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:18.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:18.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:19.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:19.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:19.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:19.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:20.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:45:20.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:20.247274+0000) 2019-10-18T03:45:20.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:20.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:20.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:20.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:20.877 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:45:20.870+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:20.878120+0000) 2019-10-18T03:45:21.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:21.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:21.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:21.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:21.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:21.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:22.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:22.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:22.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:23.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:23.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:23.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:23.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:23.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:45:23.862+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:23.866160+0000) 2019-10-18T03:45:24.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:24.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:24.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:24.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:25.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:25.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:25.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:25.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:25.715 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:45:25.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:25.716406+0000) 2019-10-18T03:45:26.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:26.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:26.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:26.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:26.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:26.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:27.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:27.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:27.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:27.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:28.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:28.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:28.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:28.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:29.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:29.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:29.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:29.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:30.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:45:30.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:30.247374+0000) 2019-10-18T03:45:30.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:30.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:30.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:30.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:30.877 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:45:30.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:30.878322+0000) 2019-10-18T03:45:31.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:31.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:31.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:31.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:31.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:31.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:32.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:32.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:32.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:32.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:33.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:33.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:33.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:45:33.866+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:33.866393+0000) 2019-10-18T03:45:34.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:34.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:34.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:34.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:35.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:35.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:35.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:35.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:35.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:45:35.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:35.716513+0000) 2019-10-18T03:45:36.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:36.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:36.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:36.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:36.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:36.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:37.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:37.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:37.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:37.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:38.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:38.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:38.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:38.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:39.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:39.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:39.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:39.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:40.246 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:45:40.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:40.247475+0000) 2019-10-18T03:45:40.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:40.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:40.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:40.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:40.877 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:45:40.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:40.878501+0000) 2019-10-18T03:45:41.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:41.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:41.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:41.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:41.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:41.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:42.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:42.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:42.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:42.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:43.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:43.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:43.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:43.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:43.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:45:43.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:43.866584+0000) 2019-10-18T03:45:44.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:44.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:44.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:44.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:45.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:45.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:45.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:45.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:45.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:45:45.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:45.716615+0000) 2019-10-18T03:45:46.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:46.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:46.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:46.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:46.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:46.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:47.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:47.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:47.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:47.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:48.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:48.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:48.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:48.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:49.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:49.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:49.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:49.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:50.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:45:50.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:50.247587+0000) 2019-10-18T03:45:50.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:50.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:50.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:50.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:50.877 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:45:50.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:50.878654+0000) 2019-10-18T03:45:51.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:51.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:51.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:51.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:51.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:51.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:52.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:52.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:52.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:52.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:53.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:53.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:53.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:53.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:53.865 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:45:53.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:53.866776+0000) 2019-10-18T03:45:54.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:54.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:54.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:54.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:55.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:55.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:55.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:55.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:55.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:45:55.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:45:55.716734+0000) 2019-10-18T03:45:56.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:56.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:56.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:56.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:56.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:45:56.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:45:57.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:57.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:57.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:57.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:58.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:58.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:58.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:58.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:45:59.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:59.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:45:59.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:45:59.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:00.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:46:00.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:00.247683+0000) 2019-10-18T03:46:00.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:00.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:00.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:00.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:00.878 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:46:00.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:00.878874+0000) 2019-10-18T03:46:01.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:01.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:01.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:01.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:01.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:01.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:02.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:02.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:02.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:02.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:03.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:03.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:03.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:03.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:03.866 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:46:03.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:03.866956+0000) 2019-10-18T03:46:04.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:04.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:04.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:04.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:05.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:05.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:05.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:05.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:05.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:46:05.712+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:05.716836+0000) 2019-10-18T03:46:06.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:06.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:06.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:06.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:06.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:06.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:07.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:07.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:07.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:07.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:08.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:08.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:08.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:08.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:09.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:09.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:09.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:09.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:10.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:46:10.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:10.247785+0000) 2019-10-18T03:46:10.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:10.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:10.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:10.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:10.878 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:46:10.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:10.879092+0000) 2019-10-18T03:46:11.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:11.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:11.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:11.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:11.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:11.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:12.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:12.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:12.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:12.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:13.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:13.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:13.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:13.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:13.866 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:46:13.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:13.867147+0000) 2019-10-18T03:46:14.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:14.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:14.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:14.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:15.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:15.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:15.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:15.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:15.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:46:15.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:15.716943+0000) 2019-10-18T03:46:16.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:16.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:16.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:16.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:16.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:16.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:17.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:17.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:17.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:17.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:18.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:18.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:18.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:18.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:19.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:19.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:19.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:19.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:20.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:46:20.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:20.247884+0000) 2019-10-18T03:46:20.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:20.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:20.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:20.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:20.878 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:46:20.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:20.879244+0000) 2019-10-18T03:46:21.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:21.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:21.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:21.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:21.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:21.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:22.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:22.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:22.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:22.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:23.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:23.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:23.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:23.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:23.866 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:46:23.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:23.867360+0000) 2019-10-18T03:46:24.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:24.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:24.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:24.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:25.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:25.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:25.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:25.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:25.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:46:25.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:25.717049+0000) 2019-10-18T03:46:26.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:26.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:26.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:26.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:26.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:26.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:27.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:27.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:27.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:27.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:28.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:28.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:28.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:28.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:29.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:29.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:29.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:29.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:30.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:30.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:30.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:30.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:30.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:46:30.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:30.247982+0000) 2019-10-18T03:46:30.878 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:46:30.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:30.879436+0000) 2019-10-18T03:46:31.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:31.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:31.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:31.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:31.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:31.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:32.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:32.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:32.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:32.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:33.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:33.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:33.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:33.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:33.866 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:46:33.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:33.867539+0000) 2019-10-18T03:46:34.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:34.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:34.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:34.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:35.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:35.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:35.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:35.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:35.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:46:35.711+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:35.717156+0000) 2019-10-18T03:46:36.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:36.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:36.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:36.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:36.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:36.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:37.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:37.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:37.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:37.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:38.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:38.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:38.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:38.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:39.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:39.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:39.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:39.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:40.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:40.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:40.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:40.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:40.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:46:40.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:40.248089+0000) 2019-10-18T03:46:40.878 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:46:40.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:40.879616+0000) 2019-10-18T03:46:41.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:41.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:41.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:41.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:41.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:41.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:42.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:42.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:42.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:42.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:43.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:43.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:43.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:43.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:43.866 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:46:43.865+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:43.867721+0000) 2019-10-18T03:46:44.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:44.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:44.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:44.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:45.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:45.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:45.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:45.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:45.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:46:45.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:45.717254+0000) 2019-10-18T03:46:46.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:46.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:46.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:46.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:46.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:46.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:47.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:47.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:47.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:47.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:48.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:48.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:48.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:48.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:49.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:49.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:49.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:49.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:50.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:50.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:50.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:50.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:50.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:46:50.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:50.248188+0000) 2019-10-18T03:46:50.878 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:46:50.877+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:50.879755+0000) 2019-10-18T03:46:51.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:51.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:51.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:51.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:51.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:51.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:52.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:52.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:52.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:52.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:53.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:53.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:53.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:53.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:53.867 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:46:53.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:53.867907+0000) 2019-10-18T03:46:54.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:54.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:54.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:54.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:55.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:55.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:55.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:55.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:55.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:46:55.710+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:46:55.717358+0000) 2019-10-18T03:46:56.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:56.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:56.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:56.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:56.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:46:56.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:46:57.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:57.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:57.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:57.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:58.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:58.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:58.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:58.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:46:59.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:59.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:46:59.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:46:59.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:00.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:00.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:00.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:00.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:00.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:47:00.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:00.248284+0000) 2019-10-18T03:47:00.879 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:47:00.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:00.879962+0000) 2019-10-18T03:47:01.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:01.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:01.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:01.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:01.966 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:01.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:02.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:02.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:02.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:02.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:03.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:03.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:03.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:03.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:03.867 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:47:03.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:03.868118+0000) 2019-10-18T03:47:04.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:04.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:04.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:04.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:05.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:05.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:05.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:05.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:05.716 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:47:05.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:05.717463+0000) 2019-10-18T03:47:06.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:06.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:06.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:06.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:06.966 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:06.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:07.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:07.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:08.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:08.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:08.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:08.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:09.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:09.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:09.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:09.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:10.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:10.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:10.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:10.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:10.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:47:10.242+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:10.248386+0000) 2019-10-18T03:47:10.879 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:47:10.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:10.880144+0000) 2019-10-18T03:47:11.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:11.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:11.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:11.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:11.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:11.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:12.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:12.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:12.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:12.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:13.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:13.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:13.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:13.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:13.867 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:47:13.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:13.868309+0000) 2019-10-18T03:47:14.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:14.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:14.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:14.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:15.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:15.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:15.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:15.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:15.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:47:15.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:15.717563+0000) 2019-10-18T03:47:16.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:16.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:16.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:16.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:16.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:16.961+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:17.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:17.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:17.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:17.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:18.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:18.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:18.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:18.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:19.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:19.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:19.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:19.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:20.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:20.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:20.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:20.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:20.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:47:20.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:20.248490+0000) 2019-10-18T03:47:20.879 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:47:20.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:20.880334+0000) 2019-10-18T03:47:21.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:21.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:21.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:21.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:21.969 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:21.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:22.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:22.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:22.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:22.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:23.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:23.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:23.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:23.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:23.867 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:47:23.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:23.868502+0000) 2019-10-18T03:47:24.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:24.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:24.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:24.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:25.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:25.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:25.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:25.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:25.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:47:25.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:25.717660+0000) 2019-10-18T03:47:26.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:26.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:26.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:26.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:26.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:26.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:27.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:27.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:27.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:27.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:28.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:28.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:28.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:28.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:29.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:29.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:29.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:29.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:30.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:30.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:30.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:30.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:30.247 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:47:30.241+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:30.248596+0000) 2019-10-18T03:47:30.879 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:47:30.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:30.880549+0000) 2019-10-18T03:47:31.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:31.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:31.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:31.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:31.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:31.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:32.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:32.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:32.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:32.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:33.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:33.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:33.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:33.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:33.867 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:47:33.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:33.868686+0000) 2019-10-18T03:47:34.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:34.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:34.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:34.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:35.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:35.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:35.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:35.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:35.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:47:35.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:35.717763+0000) 2019-10-18T03:47:36.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:36.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:36.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:36.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:36.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:36.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:37.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:37.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:37.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:37.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:38.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:38.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:38.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:39.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:39.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:39.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:39.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:40.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:40.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:40.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:40.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:40.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:47:40.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:40.248697+0000) 2019-10-18T03:47:40.879 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:47:40.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:40.880757+0000) 2019-10-18T03:47:41.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:41.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:41.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:41.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:41.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:41.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:42.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:42.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:43.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:43.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:43.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:43.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:43.867 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:47:43.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:43.868894+0000) 2019-10-18T03:47:44.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:44.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:44.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:44.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:45.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:45.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:45.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:45.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:45.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:47:45.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:45.717852+0000) 2019-10-18T03:47:46.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:46.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:46.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:46.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:46.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:46.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:47.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:47.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:47.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:47.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:48.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:48.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:48.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:48.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:49.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:49.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:50.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:50.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:50.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:50.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:50.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:47:50.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:50.248798+0000) 2019-10-18T03:47:50.880 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:47:50.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:50.880940+0000) 2019-10-18T03:47:51.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:51.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:51.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:51.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:51.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:51.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:52.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:52.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:52.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:52.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:53.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:53.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:53.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:53.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:53.868 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:47:53.864+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:53.869070+0000) 2019-10-18T03:47:54.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:54.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:54.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:54.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:55.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:55.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:55.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:47:55.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:47:55.717956+0000) 2019-10-18T03:47:56.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:56.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:56.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:56.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:56.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:47:56.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:47:57.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:57.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:57.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:57.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:58.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:58.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:58.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:58.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:47:59.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:59.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:47:59.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:47:59.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:00.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:00.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:00.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:48:00.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:00.248894+0000) 2019-10-18T03:48:00.880 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:48:00.876+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:00.881154+0000) 2019-10-18T03:48:01.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:01.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:01.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:01.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:01.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:01.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:02.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:02.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:02.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:02.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:03.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:03.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:03.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:03.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:03.868 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:48:03.863+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:03.869282+0000) 2019-10-18T03:48:04.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:04.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:04.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:04.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:05.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:05.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:05.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:05.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:05.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:48:05.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:05.718053+0000) 2019-10-18T03:48:06.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:06.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:06.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:06.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:06.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:06.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:07.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:07.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:07.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:07.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:08.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:08.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:08.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:08.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:09.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:09.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:09.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:09.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:10.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:10.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:10.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:10.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:10.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:48:10.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:10.249000+0000) 2019-10-18T03:48:10.880 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:48:10.875+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:10.881343+0000) 2019-10-18T03:48:11.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:11.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:11.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:11.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:11.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:11.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:12.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:12.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:12.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:12.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:13.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:13.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:13.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:13.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:13.868 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:48:13.863+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:13.869493+0000) 2019-10-18T03:48:14.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:14.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:14.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:14.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:15.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:15.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:15.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:15.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:15.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:48:15.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:15.718156+0000) 2019-10-18T03:48:16.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:16.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:16.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:16.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:16.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:16.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:17.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:17.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:17.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:17.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:18.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:18.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:18.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:18.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:19.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:19.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:19.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:19.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:20.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:20.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:20.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:20.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:48:20.244+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:20.249101+0000) 2019-10-18T03:48:20.880 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:48:20.875+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:20.881557+0000) 2019-10-18T03:48:21.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:21.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:21.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:21.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:21.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:21.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:22.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:22.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:22.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:22.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:23.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:23.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:23.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:23.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:23.868 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:48:23.863+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:23.869699+0000) 2019-10-18T03:48:24.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:24.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:24.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:24.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:25.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:25.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:25.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:48:25.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:25.718260+0000) 2019-10-18T03:48:26.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:26.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:26.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:26.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:26.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:26.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:27.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:27.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:27.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:27.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:28.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:28.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:28.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:28.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:29.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:29.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:29.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:29.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:30.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:30.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:30.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:30.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:30.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:48:30.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:30.249199+0000) 2019-10-18T03:48:30.881 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:48:30.875+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:30.881748+0000) 2019-10-18T03:48:31.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:31.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:31.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:31.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:31.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:31.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:32.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:32.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:32.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:32.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:33.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:33.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:33.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:33.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:33.869 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:48:33.863+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:33.869890+0000) 2019-10-18T03:48:34.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:34.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:34.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:34.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:35.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:35.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:35.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:35.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:35.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:48:35.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:35.718367+0000) 2019-10-18T03:48:36.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:36.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:36.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:36.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:36.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:36.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:37.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:37.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:37.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:37.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:38.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:38.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:39.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:39.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:39.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:39.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:40.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:40.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:40.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:40.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:40.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:48:40.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:40.249306+0000) 2019-10-18T03:48:40.881 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:48:40.875+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:40.881933+0000) 2019-10-18T03:48:41.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:41.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:41.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:41.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:41.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:41.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:42.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:42.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:42.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:42.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:43.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:43.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:43.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:43.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:43.869 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:48:43.863+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:43.870097+0000) 2019-10-18T03:48:44.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:44.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:44.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:44.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:45.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:45.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:45.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:45.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:45.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:48:45.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:45.718467+0000) 2019-10-18T03:48:46.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:46.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:46.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:46.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:46.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:46.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:47.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:47.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:47.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:47.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:48.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:48.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:48.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:48.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:49.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:49.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:49.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:49.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:50.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:50.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:50.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:50.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:48:50.243+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:50.249410+0000) 2019-10-18T03:48:50.881 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:48:50.875+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:50.882115+0000) 2019-10-18T03:48:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:51.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:51.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:51.984 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:51.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:52.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:52.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:52.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:52.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:53.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:53.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:53.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:53.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:53.869 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:48:53.867+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:53.870286+0000) 2019-10-18T03:48:54.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:54.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:54.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:54.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:55.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:55.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:55.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:55.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:55.717 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:48:55.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:48:55.718571+0000) 2019-10-18T03:48:56.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:56.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:56.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:56.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:56.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:48:56.979+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:48:57.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:57.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:57.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:57.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:58.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:58.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:58.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:58.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:59.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:59.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:59.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:59.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:48:59.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:59.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:48:59.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:48:59.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:00.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:49:00.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:00.249518+0000) 2019-10-18T03:49:00.881 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:49:00.879+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:00.882291+0000) 2019-10-18T03:49:01.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:01.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:01.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:01.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:01.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:01.982+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:02.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:02.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:02.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:02.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:03.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:03.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:03.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:03.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:03.870 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:49:03.867+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:03.870460+0000) 2019-10-18T03:49:04.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:04.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:04.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:04.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:05.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:05.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:05.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:05.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:05.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:49:05.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:05.718673+0000) 2019-10-18T03:49:06.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:06.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:06.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:06.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:06.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:06.982+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:07.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:07.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:07.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:07.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:08.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:08.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:08.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:08.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:09.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:09.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:09.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:09.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:10.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:10.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:10.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:10.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:10.248 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:49:10.246+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:10.249615+0000) 2019-10-18T03:49:10.881 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:49:10.879+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:10.882482+0000) 2019-10-18T03:49:11.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:11.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:11.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:11.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:11.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:11.982+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:12.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:12.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:13.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:13.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:13.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:13.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:13.869 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:49:13.867+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:13.870612+0000) 2019-10-18T03:49:14.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:14.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:14.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:14.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:15.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:15.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:15.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:15.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:15.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:49:15.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:15.718774+0000) 2019-10-18T03:49:16.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:16.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:16.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:16.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:16.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:16.982+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:17.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:17.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:17.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:17.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:18.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:18.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:18.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:18.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:19.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:19.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:19.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:19.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:20.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:49:20.246+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:20.249715+0000) 2019-10-18T03:49:20.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:20.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:20.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:20.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:20.881 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:49:20.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:20.882623+0000) 2019-10-18T03:49:21.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:21.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:21.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:21.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:21.989 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:21.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:22.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:22.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:22.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:22.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:23.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:23.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:23.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:23.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:23.869 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:49:23.866+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:23.870823+0000) 2019-10-18T03:49:24.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:24.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:24.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:24.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:25.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:25.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:25.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:25.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:25.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:49:25.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:25.718873+0000) 2019-10-18T03:49:26.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:26.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:26.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:26.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:26.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:26.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:27.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:27.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:27.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:27.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:28.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:28.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:28.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:28.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:29.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:29.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:29.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:29.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:30.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:49:30.246+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:30.249819+0000) 2019-10-18T03:49:30.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:30.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:30.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:30.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:30.882 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:49:30.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:30.882804+0000) 2019-10-18T03:49:31.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:31.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:31.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:31.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:31.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:31.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:32.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:32.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:32.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:32.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:33.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:33.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:33.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:33.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:33.870 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:49:33.866+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:33.871036+0000) 2019-10-18T03:49:34.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:34.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:34.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:34.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:35.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:35.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:35.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:35.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:35.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:49:35.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:35.718971+0000) 2019-10-18T03:49:36.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:36.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:36.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:36.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:36.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:36.985+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:37.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:37.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:37.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:37.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:38.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:38.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:38.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:38.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:39.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:39.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:39.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:39.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:40.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:49:40.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:40.249917+0000) 2019-10-18T03:49:40.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:40.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:40.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:40.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:40.882 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:49:40.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:40.882984+0000) 2019-10-18T03:49:41.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:41.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:41.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:41.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:41.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:41.985+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:42.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:42.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:42.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:42.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:43.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:43.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:43.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:43.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:43.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:49:43.866+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:43.871235+0000) 2019-10-18T03:49:44.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:44.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:44.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:44.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:45.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:45.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:45.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:45.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:45.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:49:45.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:45.719071+0000) 2019-10-18T03:49:46.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:46.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:46.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:46.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:46.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:46.989+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:47.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:47.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:47.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:47.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:48.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:48.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:48.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:48.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:49.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:49.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:49.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:49.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:50.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:49:50.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:50.250016+0000) 2019-10-18T03:49:50.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:50.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:50.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:50.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:50.882 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:49:50.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:50.883167+0000) 2019-10-18T03:49:51.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:51.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:51.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:51.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:51.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:51.989+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:52.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:52.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:52.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:52.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:53.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:53.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:53.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:53.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:53.873 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:49:53.866+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:53.871424+0000) 2019-10-18T03:49:54.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:54.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:54.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:54.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:55.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:55.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:55.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:55.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:55.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:49:55.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:49:55.719174+0000) 2019-10-18T03:49:56.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:56.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:56.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:56.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:56.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:49:56.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:49:57.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:57.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:57.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:57.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:58.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:58.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:58.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:58.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:49:59.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:59.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:49:59.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:49:59.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:00.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:50:00.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:00.250121+0000) 2019-10-18T03:50:00.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:00.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:00.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:00.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:00.882 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:50:00.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:00.883346+0000) 2019-10-18T03:50:01.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:01.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:01.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:01.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:01.995 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:01.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:02.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:02.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:02.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:02.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:03.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:03.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:03.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:03.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:03.870 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:50:03.866+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:03.871607+0000) 2019-10-18T03:50:04.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:04.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:04.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:04.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:05.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:05.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:05.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:05.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:05.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:50:05.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:05.719276+0000) 2019-10-18T03:50:06.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:06.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:06.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:06.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:06.996 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:06.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:07.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:07.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:07.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:07.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:08.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:08.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:08.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:08.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:09.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:09.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:09.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:09.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:10.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:50:10.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:10.250224+0000) 2019-10-18T03:50:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:10.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:10.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:10.883 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:50:10.878+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:10.883530+0000) 2019-10-18T03:50:11.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:11.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:11.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:11.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:11.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:11.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:12.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:12.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:12.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:12.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:13.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:13.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:13.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:13.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:13.875 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:50:13.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:13.871793+0000) 2019-10-18T03:50:14.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:14.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:14.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:14.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:15.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:15.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:15.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:15.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:15.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:50:15.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:15.719377+0000) 2019-10-18T03:50:16.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:16.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:16.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:16.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:16.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:16.996+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:17.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:17.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:17.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:17.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:18.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:18.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:18.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:18.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:19.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:19.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:19.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:19.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:20.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:50:20.248+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:20.250327+0000) 2019-10-18T03:50:20.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:20.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:20.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:20.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:20.882 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:50:20.882+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:20.883719+0000) 2019-10-18T03:50:21.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:21.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:21.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:21.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:21.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:21.996+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:22.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:22.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:22.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:22.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:23.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:23.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:23.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:23.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:23.871 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:50:23.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:23.871972+0000) 2019-10-18T03:50:24.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:24.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:24.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:24.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:25.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:25.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:25.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:25.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:25.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:50:25.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:25.719478+0000) 2019-10-18T03:50:26.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:26.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:26.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:26.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:26.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:26.996+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:27.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:27.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:27.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:27.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:28.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:28.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:28.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:28.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:29.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:29.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:29.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:29.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:30.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:50:30.248+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:30.250426+0000) 2019-10-18T03:50:30.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:30.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:30.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:30.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:30.883 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:50:30.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:30.883901+0000) 2019-10-18T03:50:31.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:31.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:31.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:31.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:32.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:31.996+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:32.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:32.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:32.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:32.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:33.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:33.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:33.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:33.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:33.871 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:50:33.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:33.872115+0000) 2019-10-18T03:50:34.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:34.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:34.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:34.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:35.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:50:35.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:35.719576+0000) 2019-10-18T03:50:35.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:35.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:35.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:35.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:36.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:36.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:36.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:36.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:37.001 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:37.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:37.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:37.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:37.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:37.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:38.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:38.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:38.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:38.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:39.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:39.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:39.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:39.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:40.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:50:40.248+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:40.250525+0000) 2019-10-18T03:50:40.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:40.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:40.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:40.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:40.883 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:50:40.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:40.884107+0000) 2019-10-18T03:50:41.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:41.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:41.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:41.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:42.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:42.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:42.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:42.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:42.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:42.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:43.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:43.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:43.871 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:50:43.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:43.872294+0000) 2019-10-18T03:50:44.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:44.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:44.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:44.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:45.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:45.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:45.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:45.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:45.718 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:50:45.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:45.719690+0000) 2019-10-18T03:50:46.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:46.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:46.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:46.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:47.003 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:47.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:47.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:47.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:47.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:47.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:48.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:48.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:48.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:48.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:49.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:49.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:49.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:49.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:50.249 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:50:50.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:50.250626+0000) 2019-10-18T03:50:50.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:50.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:50.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:50.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:50.883 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:50:50.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:50.884291+0000) 2019-10-18T03:50:51.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:51.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:51.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:51.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:52.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:51.999+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:52.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:52.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:52.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:52.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:53.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:53.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:53.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:53.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:53.871 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:50:53.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:53.872483+0000) 2019-10-18T03:50:54.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:54.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:54.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:54.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:55.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:55.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:55.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:55.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:55.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:50:55.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:50:55.719794+0000) 2019-10-18T03:50:56.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:56.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:56.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:56.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:57.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:50:56.999+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:50:57.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:57.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:57.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:57.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:58.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:58.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:58.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:58.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:50:59.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:59.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:50:59.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:50:59.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:00.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:51:00.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:00.250730+0000) 2019-10-18T03:51:00.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:00.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:00.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:00.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:00.889 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:51:00.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:00.884475+0000) 2019-10-18T03:51:01.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:01.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:01.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:01.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:02.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:02.003+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:02.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:02.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:02.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:02.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:03.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:03.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:03.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:03.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:03.871 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:51:03.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:03.872666+0000) 2019-10-18T03:51:04.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:04.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:04.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:04.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:05.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:05.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:05.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:05.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:05.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:51:05.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:05.719911+0000) 2019-10-18T03:51:06.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:06.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:06.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:06.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:07.007 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:07.003+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:07.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:07.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:07.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:07.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:08.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:08.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:08.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:08.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:09.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:09.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:10.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:51:10.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:10.250830+0000) 2019-10-18T03:51:10.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:10.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:10.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:10.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:10.884 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:51:10.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:10.884664+0000) 2019-10-18T03:51:11.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:11.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:11.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:11.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:12.007 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:12.003+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:12.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:12.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:12.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:12.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:13.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:13.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:13.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:13.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:13.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:51:13.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:13.872844+0000) 2019-10-18T03:51:14.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:14.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:14.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:14.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:15.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:15.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:15.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:15.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:15.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:51:15.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:15.720009+0000) 2019-10-18T03:51:16.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:16.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:16.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:16.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:17.007 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:17.003+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:17.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:17.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:17.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:17.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:18.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:18.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:18.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:18.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:19.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:19.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:19.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:19.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:20.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:51:20.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:20.250932+0000) 2019-10-18T03:51:20.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:20.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:20.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:20.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:20.883 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:51:20.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:20.884848+0000) 2019-10-18T03:51:21.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:21.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:21.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:21.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:22.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:22.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:22.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:22.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:22.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:23.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:23.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:23.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:23.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:23.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:51:23.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:23.873037+0000) 2019-10-18T03:51:24.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:24.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:24.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:24.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:25.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:25.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:25.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:25.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:25.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:51:25.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:25.720109+0000) 2019-10-18T03:51:26.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:26.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:26.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:26.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:27.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:27.006+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:27.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:27.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:27.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:27.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:28.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:28.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:28.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:28.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:29.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:29.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:29.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:29.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:30.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:51:30.246+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:30.251032+0000) 2019-10-18T03:51:30.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:30.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:30.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:30.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:30.884 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:51:30.881+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:30.885030+0000) 2019-10-18T03:51:31.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:31.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:31.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:31.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:32.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:32.006+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:32.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:32.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:32.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:32.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:33.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:33.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:33.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:33.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:33.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:51:33.869+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:33.873249+0000) 2019-10-18T03:51:34.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:34.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:34.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:34.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:35.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:35.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:35.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:51:35.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:35.720213+0000) 2019-10-18T03:51:36.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:36.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:36.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:36.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:37.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:37.006+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:37.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:37.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:37.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:37.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:38.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:38.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:38.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:38.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:39.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:39.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:39.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:39.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:40.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:51:40.246+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:40.251128+0000) 2019-10-18T03:51:40.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:40.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:40.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:40.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:40.884 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:51:40.880+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:40.885208+0000) 2019-10-18T03:51:41.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:41.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:41.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:41.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:42.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:42.006+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:42.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:42.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:42.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:42.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:43.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:43.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:43.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:43.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:43.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:51:43.868+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:43.873427+0000) 2019-10-18T03:51:44.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:44.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:44.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:44.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:45.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:45.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:45.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:45.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:45.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:51:45.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:45.720313+0000) 2019-10-18T03:51:46.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:46.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:46.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:46.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:47.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:47.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:47.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:47.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:47.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:47.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:48.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:48.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:48.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:48.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:49.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:49.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:49.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:49.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:50.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:51:50.246+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:50.251230+0000) 2019-10-18T03:51:50.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:50.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:50.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:50.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:50.884 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:51:50.880+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:50.885400+0000) 2019-10-18T03:51:51.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:51.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:51.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:51.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:52.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:52.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:52.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:52.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:52.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:52.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:53.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:53.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:53.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:53.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:53.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:51:53.868+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:53.873610+0000) 2019-10-18T03:51:54.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:54.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:54.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:54.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:55.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:55.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:55.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:55.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:55.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:51:55.714+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:51:55.720415+0000) 2019-10-18T03:51:56.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:56.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:56.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:56.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:57.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:51:57.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:51:57.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:57.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:57.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:57.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:58.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:58.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:58.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:58.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:51:59.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:59.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:51:59.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:51:59.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:00.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:52:00.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:00.251336+0000) 2019-10-18T03:52:00.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:00.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:00.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:00.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:00.885 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:00.880+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:00.885613+0000) 2019-10-18T03:52:01.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:01.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:01.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:01.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:02.015 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:02.009+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:02.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:02.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:02.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:02.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:03.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:03.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:03.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:03.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:03.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:52:03.868+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:03.873781+0000) 2019-10-18T03:52:04.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:04.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:04.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:04.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:05.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:05.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:05.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:05.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:05.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:52:05.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:05.720514+0000) 2019-10-18T03:52:06.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:06.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:06.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:06.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:07.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:07.009+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:07.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:07.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:07.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:07.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:08.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:08.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:08.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:08.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:09.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:09.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:09.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:09.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:10.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:52:10.245+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:10.251439+0000) 2019-10-18T03:52:10.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:10.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:10.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:10.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:10.884 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:10.880+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:10.885802+0000) 2019-10-18T03:52:11.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:11.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:11.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:11.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:12.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:12.013+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:12.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:12.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:12.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:12.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:13.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:13.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:13.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:13.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:13.873 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:52:13.868+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:13.873989+0000) 2019-10-18T03:52:14.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:14.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:14.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:14.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:15.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:15.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:15.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:15.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:15.719 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:52:15.713+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:15.720614+0000) 2019-10-18T03:52:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:16.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:16.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:16.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:17.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:17.013+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:17.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:17.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:17.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:17.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:18.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:18.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:18.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:18.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:19.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:19.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:19.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:19.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:20.250 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:52:20.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:20.251546+0000) 2019-10-18T03:52:20.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:20.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:20.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:20.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:20.885 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:20.880+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:20.885984+0000) 2019-10-18T03:52:21.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:21.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:21.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:21.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:22.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:22.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:22.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:22.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:22.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:22.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:23.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:23.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:23.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:23.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:23.873 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:52:23.872+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:23.874175+0000) 2019-10-18T03:52:24.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:24.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:24.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:24.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:25.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:25.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:25.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:25.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:25.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:52:25.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:25.720715+0000) 2019-10-18T03:52:26.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:26.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:26.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:26.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:27.019 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:27.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:27.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:27.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:27.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:27.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:28.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:28.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:28.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:28.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:29.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:29.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:29.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:29.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:30.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:52:30.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:30.251647+0000) 2019-10-18T03:52:30.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:30.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:30.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:30.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:30.885 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:30.884+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:30.886160+0000) 2019-10-18T03:52:30.886 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:30.884+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:30.887333+0000) 2019-10-18T03:52:31.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:31.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:31.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:31.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:32.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:32.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:32.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:32.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:32.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:32.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:33.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:33.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:33.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:33.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:33.873 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:52:33.872+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:33.874352+0000) 2019-10-18T03:52:34.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:34.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:34.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:34.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:35.385 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:35.384+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:35.386673+0000) 2019-10-18T03:52:35.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:35.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:35.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:35.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:35.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:52:35.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:35.720815+0000) 2019-10-18T03:52:36.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:36.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:36.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:36.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:37.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:37.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:37.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:37.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:37.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:37.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:38.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:38.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:38.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:38.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:39.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:39.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:39.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:39.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:40.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:52:40.248+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:40.251746+0000) 2019-10-18T03:52:40.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:40.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:40.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:40.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:41.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:41.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:41.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:41.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:42.022 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:42.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:42.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:42.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:42.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:42.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:43.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:43.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:43.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:43.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:43.873 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:52:43.872+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:43.874535+0000) 2019-10-18T03:52:44.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:44.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:44.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:44.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:45.386 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:45.384+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:45.386859+0000) 2019-10-18T03:52:45.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:45.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:45.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:45.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:45.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:52:45.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:45.720914+0000) 2019-10-18T03:52:46.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:46.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:46.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:46.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:47.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:47.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:47.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:47.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:47.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:47.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:48.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:48.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:48.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:48.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:49.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:49.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:49.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:50.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:52:50.248+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:50.251852+0000) 2019-10-18T03:52:50.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:50.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:50.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:50.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:51.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:51.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:51.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:51.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:52.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:52.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:52.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:52.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:52.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:52.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:53.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:53.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:53.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:53.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:53.874 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:52:53.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:53.874752+0000) 2019-10-18T03:52:54.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:54.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:54.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:54.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:55.386 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:52:55.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:55.387081+0000) 2019-10-18T03:52:55.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:55.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:55.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:55.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:55.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:52:55.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:52:55.721009+0000) 2019-10-18T03:52:56.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:56.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:56.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:56.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:57.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:52:57.020+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:52:57.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:57.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:57.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:57.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:58.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:58.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:58.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:58.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 372 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:52:59.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:59.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:52:59.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:52:59.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:00.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:53:00.248+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:00.251954+0000) 2019-10-18T03:53:00.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:00.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:00.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:00.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:01.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:01.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:01.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:01.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:02.025 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:02.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:02.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:02.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:02.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:02.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:03.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:03.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:03.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:03.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:03.874 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:53:03.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:03.874961+0000) 2019-10-18T03:53:04.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:04.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:04.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:04.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:05.386 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:53:05.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:05.387267+0000) 2019-10-18T03:53:05.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:05.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:05.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:05.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:05.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:53:05.716+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:05.721112+0000) 2019-10-18T03:53:06.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:06.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:06.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:06.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:07.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:07.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:07.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:07.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:07.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:07.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:08.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:08.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:08.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:08.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:09.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:09.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:09.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:09.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:10.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:53:10.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:10.252056+0000) 2019-10-18T03:53:10.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:10.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:10.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:10.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:11.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:11.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:11.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:11.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:12.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:12.023+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:12.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:12.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:12.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:12.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:13.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:13.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:13.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:13.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:13.874 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:53:13.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:13.875155+0000) 2019-10-18T03:53:14.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:14.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:14.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:14.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:15.386 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:53:15.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:15.387471+0000) 2019-10-18T03:53:15.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:15.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:15.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:15.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:15.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:53:15.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:15.721213+0000) 2019-10-18T03:53:16.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:16.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:16.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:16.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:17.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:17.023+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:17.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:17.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:17.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:17.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:18.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:18.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:18.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:18.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:19.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:19.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:19.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:19.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:20.251 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:53:20.247+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:20.252162+0000) 2019-10-18T03:53:20.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:20.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:20.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:20.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:21.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:21.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:21.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:21.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:22.029 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:22.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:22.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:22.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:22.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:22.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:23.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:23.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:23.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:23.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:23.874 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:53:23.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:23.875370+0000) 2019-10-18T03:53:24.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:24.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:24.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:24.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:25.386 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:53:25.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:25.387659+0000) 2019-10-18T03:53:25.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:53:25.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:25.721312+0000) 2019-10-18T03:53:25.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:25.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:25.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:25.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:26.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:26.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:26.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:26.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:27.029 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:27.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:27.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:27.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:27.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:27.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:28.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:28.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:28.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:28.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:29.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:29.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:42:55.244936+0000 (oldest deadline 2019-10-18T03:43:15.244936+0000) 2019-10-18T03:53:29.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:29.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:30.252 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:53:30.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:30.253497+0000) 2019-10-18T03:53:30.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:30.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:31.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:31.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:32.031 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:32.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:32.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:32.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:33.874 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:53:33.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:33.875585+0000) 2019-10-18T03:53:33.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:33.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:34.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:34.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:35.387 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:53:35.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:35.387797+0000) 2019-10-18T03:53:35.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:53:35.715+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:35.721416+0000) 2019-10-18T03:53:35.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:35.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:36.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:36.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:37.031 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:37.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:37.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:37.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:38.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:38.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:39.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:39.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:40.252 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:53:40.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:40.253597+0000) 2019-10-18T03:53:40.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:40.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:41.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:41.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:42.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:42.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:42.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:42.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:43.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:43.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:43.874 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:53:43.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:43.875772+0000) 2019-10-18T03:53:44.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:44.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:45.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:53:45.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:45.387983+0000) 2019-10-18T03:53:45.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:53:45.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:45.721524+0000) 2019-10-18T03:53:45.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:45.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:46.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:46.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:47.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:47.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:47.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:47.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:48.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:48.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:49.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:49.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:50.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:53:50.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:50.253699+0000) 2019-10-18T03:53:50.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:50.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:50.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:50.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:51.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:51.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:51.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:51.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:52.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:52.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:52.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:52.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:52.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:52.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:53.875 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:53:53.871+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:53.875963+0000) 2019-10-18T03:53:53.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:53.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:53.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:53.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:54.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:54.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:54.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:54.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:55.387 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:53:55.383+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:55.388162+0000) 2019-10-18T03:53:55.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:53:55.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:53:55.721623+0000) 2019-10-18T03:53:55.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:55.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:55.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:55.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:56.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:56.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:56.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:56.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:57.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:53:57.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:53:57.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:57.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:57.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:57.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:58.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:58.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:58.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:58.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:53:59.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:59.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:53:59.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:53:59.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:00.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:54:00.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:00.253798+0000) 2019-10-18T03:54:00.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:00.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:00.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:00.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:01.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:01.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:01.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:01.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:02.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:02.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:03.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:03.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:03.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:03.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:03.875 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:54:03.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:03.876150+0000) 2019-10-18T03:54:04.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:03.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:04.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:03.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:05.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:05.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:05.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:05.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:05.387 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:54:05.382+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:05.388340+0000) 2019-10-18T03:54:05.720 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:54:05.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:05.721723+0000) 2019-10-18T03:54:06.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:06.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:06.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:06.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:07.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:07.030+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:07.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:07.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:07.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:07.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:08.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:08.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:08.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:08.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:08.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:08.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:08.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:10.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:10.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:10.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:10.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:10.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:54:10.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:10.253898+0000) 2019-10-18T03:54:11.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:11.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:11.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:11.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:12.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:12.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:12.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:12.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:12.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:12.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:13.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:13.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:13.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:13.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:13.875 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:54:13.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:13.876339+0000) 2019-10-18T03:54:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:14.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:14.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:15.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:15.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:15.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:15.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:15.387 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:54:15.382+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:15.388519+0000) 2019-10-18T03:54:15.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:54:15.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:15.721826+0000) 2019-10-18T03:54:16.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:16.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:16.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:16.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:17.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:17.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:17.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:17.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:17.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:17.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:18.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:18.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:18.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:18.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:19.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:19.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:19.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:19.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:20.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:20.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:20.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:20.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:20.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:54:20.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:20.253999+0000) 2019-10-18T03:54:21.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:21.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:21.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:21.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:22.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:22.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:22.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:22.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:22.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:22.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:23.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:23.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:23.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:23.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:23.875 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:54:23.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:23.876577+0000) 2019-10-18T03:54:24.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:24.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:24.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:24.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:25.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:25.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:25.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:25.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:25.387 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:54:25.382+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:25.388697+0000) 2019-10-18T03:54:25.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:54:25.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:25.721926+0000) 2019-10-18T03:54:26.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:26.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:26.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:26.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:27.039 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:27.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:27.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:27.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:27.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:27.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:28.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:28.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:28.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:28.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:29.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:29.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:29.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:29.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:30.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:30.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:30.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:30.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:30.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:54:30.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:30.254098+0000) 2019-10-18T03:54:31.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:31.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:31.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:31.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:32.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:32.033+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:32.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:32.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:32.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:32.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:33.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:33.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:33.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:33.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:33.875 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:54:33.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:33.876770+0000) 2019-10-18T03:54:34.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:34.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:34.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:34.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:35.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:35.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:35.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:35.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:35.387 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:54:35.382+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:35.388841+0000) 2019-10-18T03:54:35.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:54:35.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:35.722031+0000) 2019-10-18T03:54:36.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:36.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:36.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:36.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:37.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:37.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:37.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:37.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:37.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:37.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:38.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:38.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:38.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:38.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:38.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:38.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:38.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:38.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:40.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:40.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:40.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:40.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:54:40.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:40.254198+0000) 2019-10-18T03:54:41.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:41.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:41.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:41.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:42.043 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:42.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:42.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:42.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:42.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:42.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:43.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:43.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:43.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:43.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:43.876 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:54:43.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:43.876989+0000) 2019-10-18T03:54:44.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:44.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:44.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:44.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:45.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:45.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:45.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:45.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:45.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:54:45.382+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:45.389023+0000) 2019-10-18T03:54:45.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:54:45.721+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:45.722132+0000) 2019-10-18T03:54:46.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:46.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:46.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:46.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:47.043 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:47.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:47.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:47.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:47.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:47.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:48.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:48.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:48.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:48.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:49.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:49.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:49.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:49.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:50.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:50.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:50.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:50.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:50.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:54:50.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:50.254300+0000) 2019-10-18T03:54:51.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:51.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:51.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:51.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:52.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:52.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:52.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:52.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:52.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:52.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:53.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:53.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:53.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:53.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:53.876 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:54:53.870+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:53.877210+0000) 2019-10-18T03:54:54.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:54.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:54.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:54.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:55.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:55.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:55.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:55.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:55.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:54:55.382+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:55.389206+0000) 2019-10-18T03:54:55.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:54:55.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:54:55.722229+0000) 2019-10-18T03:54:56.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:56.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:56.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:56.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:57.045 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:54:57.044+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:54:57.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:57.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:57.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:57.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:58.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:58.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:58.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:58.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:54:59.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:59.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:54:59.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:54:59.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:00.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:00.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:00.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:55:00.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:00.254399+0000) 2019-10-18T03:55:01.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:01.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:01.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:01.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:02.045 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:02.044+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:02.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:02.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:02.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:02.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:03.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:03.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:03.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:03.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:03.876 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:55:03.874+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:03.877428+0000) 2019-10-18T03:55:04.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:04.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:04.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:04.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:05.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:05.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:05.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:05.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:05.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:55:05.386+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:05.389355+0000) 2019-10-18T03:55:05.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:55:05.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:05.722330+0000) 2019-10-18T03:55:06.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:06.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:06.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:06.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:07.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:07.044+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:07.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:07.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:07.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:07.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:08.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:08.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:08.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:08.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:09.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:09.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:09.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:09.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:10.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:10.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:10.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:10.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:10.253 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:55:10.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:10.254500+0000) 2019-10-18T03:55:11.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:11.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:11.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:11.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:12.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:12.044+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:12.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:12.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:12.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:12.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:13.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:13.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:13.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:13.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:13.877 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:55:13.873+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:13.877610+0000) 2019-10-18T03:55:14.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:14.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:14.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:14.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:15.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:15.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:15.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:15.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:15.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:55:15.385+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:15.389540+0000) 2019-10-18T03:55:15.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:55:15.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:15.722433+0000) 2019-10-18T03:55:16.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:16.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:16.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:16.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:17.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:17.044+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:17.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:17.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:17.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:17.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:18.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:18.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:18.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:18.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:19.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:19.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:19.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:19.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:20.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:55:20.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:20.254603+0000) 2019-10-18T03:55:20.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:20.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:20.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:20.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:21.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:21.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:21.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:21.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:22.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:22.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:22.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:22.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:22.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:22.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:23.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:23.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:23.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:23.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:23.876 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:55:23.873+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:23.877798+0000) 2019-10-18T03:55:24.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:24.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:24.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:24.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:25.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:25.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:25.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:25.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:25.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:55:25.385+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:25.389727+0000) 2019-10-18T03:55:25.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:55:25.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:25.722532+0000) 2019-10-18T03:55:26.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:26.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:26.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:26.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:27.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:27.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:27.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:27.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:27.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:27.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:28.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:28.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:28.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:28.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:29.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:29.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:29.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:29.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:30.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:55:30.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:30.254700+0000) 2019-10-18T03:55:30.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:30.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:30.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:30.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:31.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:31.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:31.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:31.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:32.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:32.047+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:32.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:32.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:32.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:32.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:33.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:33.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:33.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:33.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:33.877 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:55:33.873+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:33.878006+0000) 2019-10-18T03:55:34.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:34.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:34.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:34.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:35.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:35.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:35.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:35.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:35.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:55:35.385+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:35.389876+0000) 2019-10-18T03:55:35.721 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:55:35.719+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:35.722628+0000) 2019-10-18T03:55:36.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:36.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:36.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:36.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:37.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:37.047+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:37.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:37.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:37.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:37.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:38.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:38.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:38.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:38.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:39.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:39.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:39.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:39.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:40.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:55:40.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:40.254801+0000) 2019-10-18T03:55:40.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:40.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:40.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:40.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:41.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:41.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:41.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:41.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:42.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:42.047+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:42.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:42.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:42.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:42.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:43.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:43.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:43.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:43.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:43.877 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:55:43.877+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:43.878211+0000) 2019-10-18T03:55:44.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:44.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:44.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:44.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:45.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:45.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:45.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:45.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:45.388 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:55:45.385+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:45.390067+0000) 2019-10-18T03:55:45.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:55:45.719+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:45.722729+0000) 2019-10-18T03:55:46.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:46.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:46.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:46.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:47.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:47.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:47.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:47.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:47.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:47.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:48.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:48.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:48.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:48.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:49.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:49.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:49.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:49.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:50.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:55:50.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:50.254901+0000) 2019-10-18T03:55:50.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:50.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:50.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:50.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:51.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:51.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:51.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:51.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:52.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:52.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:52.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:52.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:52.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:52.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:53.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:53.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:53.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:53.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:53.877 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:55:53.877+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:53.878351+0000) 2019-10-18T03:55:54.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:54.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:54.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:54.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:55.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:55.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:55.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:55.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:55.391 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:55:55.389+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:55.390202+0000) 2019-10-18T03:55:55.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:55:55.719+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:55:55.722827+0000) 2019-10-18T03:55:56.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:56.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:56.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:56.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:57.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:55:57.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:55:57.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:57.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:57.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:57.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:58.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:58.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:58.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:58.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:55:59.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:59.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:55:59.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:55:59.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:00.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:56:00.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:00.255000+0000) 2019-10-18T03:56:00.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:00.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:00.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:00.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:01.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:01.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:01.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:01.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:02.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:02.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:02.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:02.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:02.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:02.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:03.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:03.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:03.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:03.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:03.877 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:56:03.877+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:03.878534+0000) 2019-10-18T03:56:04.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:04.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:04.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:04.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:05.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:05.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:05.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:05.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:05.389 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:56:05.389+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:05.390388+0000) 2019-10-18T03:56:05.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:56:05.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:05.722925+0000) 2019-10-18T03:56:06.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:06.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:06.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:06.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:07.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:07.050+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:07.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:07.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:07.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:07.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:08.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:08.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:08.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:08.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:09.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:09.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:09.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:09.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:10.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:56:10.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:10.255105+0000) 2019-10-18T03:56:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:10.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:10.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:10.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:11.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:11.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:11.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:11.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:12.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:12.054+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:12.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:12.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:12.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:12.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:13.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:13.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:13.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:13.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:13.878 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:56:13.877+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:13.878680+0000) 2019-10-18T03:56:14.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:14.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:14.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:14.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:15.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:15.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:15.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:56:15.389+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:15.390574+0000) 2019-10-18T03:56:15.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:56:15.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:15.723026+0000) 2019-10-18T03:56:16.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:16.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:16.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:16.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:17.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:17.054+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:17.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:17.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:17.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:17.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:18.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:18.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:18.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:18.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:19.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:19.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:19.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:19.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:20.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:20.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:20.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:20.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:20.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:56:20.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:20.255206+0000) 2019-10-18T03:56:21.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:21.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:21.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:21.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:22.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:22.054+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:22.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:22.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:22.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:22.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:23.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:23.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:23.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:23.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:23.878 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:56:23.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:23.878887+0000) 2019-10-18T03:56:24.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:24.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:24.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:24.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:25.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:25.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:25.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:25.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:56:25.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:25.390783+0000) 2019-10-18T03:56:25.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:56:25.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:25.723123+0000) 2019-10-18T03:56:26.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:26.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:26.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:26.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:27.059 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:27.054+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:27.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:27.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:27.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:27.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:28.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:28.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:28.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:28.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:29.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:29.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:29.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:29.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:30.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:30.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:30.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:30.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:30.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:56:30.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:30.255305+0000) 2019-10-18T03:56:31.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:31.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:31.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:31.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:32.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:32.054+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:32.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:32.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:32.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:32.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:33.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:33.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:33.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:33.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:33.878 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:56:33.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:33.879083+0000) 2019-10-18T03:56:34.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:34.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:34.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:34.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:35.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:35.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:35.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:35.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:35.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:56:35.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:35.390961+0000) 2019-10-18T03:56:35.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:56:35.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:35.723225+0000) 2019-10-18T03:56:35.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:35.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:35.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:35.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:36.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:36.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:36.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:36.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:37.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:37.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:37.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:37.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:37.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:37.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:38.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:38.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:38.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:38.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:39.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:39.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:39.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:39.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:40.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:56:40.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:40.255405+0000) 2019-10-18T03:56:40.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:40.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:40.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:40.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:41.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:41.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:41.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:41.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:42.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:42.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:42.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:42.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:42.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:42.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:43.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:56:43.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:43.879294+0000) 2019-10-18T03:56:43.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:43.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:43.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:43.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:44.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:44.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:44.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:44.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:45.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:56:45.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:45.391144+0000) 2019-10-18T03:56:45.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:56:45.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:45.723322+0000) 2019-10-18T03:56:45.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:45.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:45.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:45.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:46.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:46.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:46.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:46.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:47.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:47.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:47.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:47.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:47.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:47.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:48.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:48.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:48.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:48.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:49.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:49.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:49.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:49.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:50.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:56:50.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:50.255505+0000) 2019-10-18T03:56:50.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:50.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:50.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:50.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:51.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:51.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:51.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:51.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:52.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:52.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:52.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:52.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:52.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:52.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:53.878 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:56:53.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:53.879486+0000) 2019-10-18T03:56:53.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:53.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:53.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:53.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 384 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:54.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:54.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:54.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:54.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:55.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:56:55.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:55.391327+0000) 2019-10-18T03:56:55.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:56:55.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:56:55.723420+0000) 2019-10-18T03:56:55.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:55.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:55.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:55.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:56.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:56.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:56.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:56.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:57.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:56:57.057+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:56:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:57.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:57.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:58.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:58.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:58.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:58.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:56:59.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:59.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:56:59.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:56:59.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:00.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:57:00.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:00.255605+0000) 2019-10-18T03:57:00.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:00.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:00.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:00.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:02.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:02.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:02.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:02.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:02.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:02.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:03.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:03.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:03.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:03.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:03.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:57:03.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:03.879700+0000) 2019-10-18T03:57:04.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:04.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:04.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:04.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:04.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:04.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:04.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:04.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:05.391 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:57:05.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:05.391504+0000) 2019-10-18T03:57:05.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:57:05.721+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:05.723524+0000) 2019-10-18T03:57:05.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:05.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:05.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:05.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:06.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:06.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:06.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:06.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:07.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:07.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:07.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:07.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:07.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:07.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:08.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:08.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:08.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:08.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:09.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:09.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:09.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:09.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:10.254 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:57:10.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:10.255705+0000) 2019-10-18T03:57:11.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:11.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:11.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:11.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:12.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:12.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:12.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:12.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:12.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:12.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:13.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:13.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:13.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:13.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:13.878 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:57:13.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:13.879884+0000) 2019-10-18T03:57:13.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:13.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:13.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:13.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:15.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:15.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:15.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:15.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:15.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:57:15.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:15.391660+0000) 2019-10-18T03:57:15.722 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:57:15.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:15.723624+0000) 2019-10-18T03:57:16.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:16.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:16.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:16.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:17.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:17.064+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:17.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:17.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:17.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:17.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:18.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:18.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:18.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:18.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:19.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:19.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:19.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:19.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:20.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:20.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:20.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:20.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:20.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:57:20.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:20.255808+0000) 2019-10-18T03:57:21.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:21.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:21.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:21.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:22.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:22.068+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:22.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:22.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:22.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:22.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:23.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:23.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:23.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:23.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:23.879 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:57:23.876+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:23.880100+0000) 2019-10-18T03:57:24.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:24.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:24.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:24.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:25.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:25.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:25.390 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:57:25.388+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:25.391824+0000) 2019-10-18T03:57:25.723 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:57:25.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:25.723721+0000) 2019-10-18T03:57:26.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:26.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:26.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:26.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:27.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:27.068+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:27.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:27.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:27.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:27.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:28.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:28.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:28.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:28.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:29.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:29.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:29.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:29.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:30.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:30.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:30.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:30.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:30.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:57:30.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:30.255905+0000) 2019-10-18T03:57:31.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:31.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:31.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:31.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:32.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:32.068+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:32.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:32.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:32.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:32.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:33.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:33.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:33.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:33.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:33.879 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:57:33.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:33.880312+0000) 2019-10-18T03:57:34.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:34.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:34.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:34.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:35.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:35.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:35.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:35.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:35.391 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:57:35.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:35.392009+0000) 2019-10-18T03:57:35.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:57:35.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:35.723831+0000) 2019-10-18T03:57:36.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:36.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:36.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:36.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:37.071 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:37.068+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:37.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:37.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:37.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:37.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:38.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:38.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:38.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:38.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:39.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:39.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:39.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:39.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:40.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:40.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:40.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:40.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:40.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:57:40.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:40.256014+0000) 2019-10-18T03:57:41.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:41.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:41.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:41.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:42.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:42.068+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:42.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:42.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:42.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:42.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:43.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:43.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:43.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:43.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:43.879 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:57:43.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:43.880496+0000) 2019-10-18T03:57:44.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:44.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:44.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:44.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:45.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:45.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:45.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:45.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:45.391 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:57:45.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:45.392188+0000) 2019-10-18T03:57:45.723 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:57:45.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:45.723941+0000) 2019-10-18T03:57:46.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:46.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:46.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:46.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:47.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:47.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:47.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:47.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:47.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:47.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:48.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:48.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:48.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:48.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:49.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:49.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:49.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:49.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:50.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:57:50.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:50.256116+0000) 2019-10-18T03:57:50.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:50.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:50.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:50.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:51.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:51.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:51.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:51.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:52.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:52.071+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:52.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:52.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:52.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:52.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:53.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:53.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:53.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:53.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:53.879 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:57:53.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:53.880687+0000) 2019-10-18T03:57:54.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:54.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:54.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:54.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:55.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:55.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:55.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:55.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:55.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:57:55.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:55.392357+0000) 2019-10-18T03:57:55.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:57:55.719+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:57:55.724038+0000) 2019-10-18T03:57:56.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:56.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:56.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:56.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:57.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:57:57.071+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:57:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:57.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:57.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:58.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:58.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:58.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:58.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:57:59.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:59.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:57:59.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:57:59.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:00.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:58:00.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:00.256214+0000) 2019-10-18T03:58:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:00.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:00.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:00.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:01.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:01.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:01.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:01.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:02.075 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:02.071+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:02.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:02.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:02.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:02.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:03.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:03.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:03.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:03.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:03.879 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:58:03.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:03.880880+0000) 2019-10-18T03:58:04.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:04.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:04.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:04.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:05.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:05.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:05.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:05.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:05.391 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:58:05.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:05.392535+0000) 2019-10-18T03:58:05.723 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:58:05.719+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:05.724138+0000) 2019-10-18T03:58:06.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:06.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:06.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:06.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:07.076 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:07.071+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:07.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:07.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:07.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:07.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:08.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:08.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:08.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:08.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:09.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:09.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:09.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:09.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:10.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:58:10.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:10.256319+0000) 2019-10-18T03:58:10.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:10.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:10.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:10.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:11.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:11.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:11.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:11.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:12.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:12.075+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:12.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:12.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:12.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:12.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:13.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:13.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:13.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:13.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:13.882 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:58:13.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:13.881090+0000) 2019-10-18T03:58:14.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:14.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:14.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:14.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:15.391 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:58:15.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:15.392723+0000) 2019-10-18T03:58:15.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:15.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:15.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:15.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:15.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:58:15.719+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:15.724244+0000) 2019-10-18T03:58:16.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:16.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:16.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:16.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:17.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:17.075+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:17.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:17.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:17.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:17.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:18.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:18.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:18.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:18.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:19.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:19.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:19.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:19.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:20.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:58:20.251+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:20.256420+0000) 2019-10-18T03:58:20.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:20.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:20.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:20.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:21.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:21.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:21.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:21.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:22.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:22.075+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:22.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:22.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:22.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:22.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:23.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:23.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:23.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:23.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:23.880 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:58:23.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:23.881284+0000) 2019-10-18T03:58:24.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:24.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:24.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:24.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:25.392 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:58:25.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:25.392901+0000) 2019-10-18T03:58:25.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:25.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:25.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:25.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:25.723 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:58:25.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:25.724346+0000) 2019-10-18T03:58:26.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:26.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:26.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:26.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:27.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:27.074+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:27.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:27.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:28.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:28.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:28.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:28.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:29.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:29.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:29.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:29.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:30.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:58:30.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:30.256518+0000) 2019-10-18T03:58:30.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:30.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:30.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:30.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:31.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:31.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:31.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:31.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:32.080 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:32.074+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:32.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:32.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:32.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:33.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:33.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:33.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:33.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:33.880 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:58:33.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:33.881496+0000) 2019-10-18T03:58:34.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:34.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:34.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:34.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:35.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:35.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:35.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:35.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:35.392 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:58:35.387+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:35.393084+0000) 2019-10-18T03:58:35.723 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:58:35.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:35.724457+0000) 2019-10-18T03:58:36.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:36.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:36.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:36.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:37.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:37.078+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:37.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:37.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:37.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:37.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:38.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:38.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:38.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:38.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:39.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:39.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:39.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:39.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:40.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:40.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:40.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:40.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:40.255 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:58:40.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:40.256621+0000) 2019-10-18T03:58:41.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:41.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:41.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:41.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:42.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:42.078+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:42.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:42.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:42.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:42.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:43.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:43.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:43.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:43.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:43.880 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:58:43.875+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:43.881685+0000) 2019-10-18T03:58:44.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:44.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:44.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:44.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:45.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:45.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:45.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:45.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:45.392 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:58:45.386+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:45.393271+0000) 2019-10-18T03:58:45.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:58:45.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:45.724557+0000) 2019-10-18T03:58:46.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:46.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:46.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:46.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:47.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:47.078+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:47.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:47.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:47.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:47.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:48.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:48.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:48.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:48.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:49.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:49.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:49.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:49.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:50.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:58:50.250+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:50.256723+0000) 2019-10-18T03:58:50.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:50.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:50.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:50.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:51.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:51.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:51.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:51.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:52.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:52.078+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:52.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:52.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:52.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:52.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:53.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:53.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:53.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:53.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:53.880 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:58:53.874+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:53.881870+0000) 2019-10-18T03:58:54.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:54.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:54.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:54.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:55.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:55.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:55.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:55.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:55.392 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:58:55.386+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:55.393445+0000) 2019-10-18T03:58:55.723 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:58:55.718+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:58:55.724654+0000) 2019-10-18T03:58:56.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:56.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:56.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:56.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:57.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:58:57.078+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:58:57.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:57.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:57.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:57.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:58.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:58.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:58.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:58.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:58:59.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:59.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:58:59.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:58:59.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:00.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:59:00.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:00.256821+0000) 2019-10-18T03:59:00.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:00.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:00.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:00.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:01.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:01.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:01.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:01.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:02.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:02.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:02.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:02.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:02.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:02.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:03.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:03.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:03.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:03.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:03.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:59:03.874+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:03.882109+0000) 2019-10-18T03:59:04.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:04.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:04.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:04.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:05.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:05.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:05.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:05.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:05.393 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:59:05.386+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:05.393630+0000) 2019-10-18T03:59:05.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:59:05.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:05.724758+0000) 2019-10-18T03:59:06.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:06.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:06.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:06.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:07.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:07.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:07.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:07.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:07.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:07.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:08.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:08.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:08.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:08.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:09.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:09.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:09.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:09.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:10.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:59:10.249+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:10.256922+0000) 2019-10-18T03:59:10.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:10.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:10.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:10.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:11.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:11.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:11.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:11.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:12.087 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:12.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:12.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:12.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:12.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:12.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:13.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:13.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:13.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:13.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:13.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:59:13.878+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:13.882252+0000) 2019-10-18T03:59:14.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:14.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:14.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:14.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:15.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:15.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:15.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:15.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:15.392 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:59:15.386+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:15.393827+0000) 2019-10-18T03:59:15.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:59:15.717+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:15.724853+0000) 2019-10-18T03:59:16.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:16.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:16.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:16.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:17.087 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:17.081+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:17.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:17.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:17.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:17.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:18.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:18.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:18.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:18.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:19.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:19.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:19.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:19.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:20.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:59:20.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:20.257024+0000) 2019-10-18T03:59:20.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:20.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:20.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:20.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:21.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:21.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:21.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:21.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:22.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:22.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:22.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:22.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:22.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:22.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:23.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:23.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:23.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:23.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:23.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:59:23.878+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:23.882465+0000) 2019-10-18T03:59:24.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:24.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:24.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:24.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:25.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:25.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:25.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:25.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:25.393 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:59:25.386+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:25.394010+0000) 2019-10-18T03:59:25.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:59:25.721+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:25.724946+0000) 2019-10-18T03:59:26.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:26.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:26.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:26.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:27.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:27.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:27.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:27.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:27.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:27.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:28.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:28.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:28.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:28.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:29.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:29.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:29.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:29.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:30.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:59:30.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:30.257131+0000) 2019-10-18T03:59:30.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:30.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:30.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:30.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:31.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:31.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:31.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:31.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:32.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:32.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:32.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:32.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:33.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:33.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:33.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:33.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:33.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:59:33.878+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:33.882647+0000) 2019-10-18T03:59:34.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:34.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:34.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:34.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:35.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:35.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:35.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:35.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:35.393 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:59:35.390+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:35.394218+0000) 2019-10-18T03:59:35.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:59:35.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:35.725046+0000) 2019-10-18T03:59:36.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:36.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:36.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:36.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:37.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:37.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:37.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:37.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:37.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:37.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:38.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:38.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:38.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:38.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:39.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:39.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:39.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:39.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:40.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:59:40.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:40.257234+0000) 2019-10-18T03:59:40.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:40.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:40.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:40.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:41.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:41.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:42.092 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:42.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:42.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:42.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:42.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:42.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:43.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:43.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:43.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:43.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:43.881 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:59:43.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:43.882830+0000) 2019-10-18T03:59:44.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:44.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:44.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:44.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:45.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:45.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:45.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:45.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:45.393 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:59:45.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:45.394401+0000) 2019-10-18T03:59:45.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:59:45.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:45.725148+0000) 2019-10-18T03:59:46.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:46.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:46.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:46.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:47.092 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:47.088+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:47.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:47.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:47.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:47.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:48.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:48.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:48.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:48.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:49.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:49.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:49.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:49.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:50.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:50.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:50.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:50.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:50.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T03:59:50.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:50.257334+0000) 2019-10-18T03:59:51.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:51.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:51.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:51.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:52.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:52.092+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:52.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:52.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:52.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:52.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:53.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:53.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:53.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:53.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:53.882 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T03:59:53.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:53.882983+0000) 2019-10-18T03:59:54.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:54.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:54.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:54.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:55.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:55.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:55.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:55.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:55.393 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T03:59:55.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:55.394578+0000) 2019-10-18T03:59:55.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T03:59:55.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T02:59:55.725246+0000) 2019-10-18T03:59:56.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:56.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:56.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:56.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:57.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T03:59:57.092+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T03:59:57.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:57.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:57.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:57.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:58.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:58.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:58.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:58.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T03:59:59.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:59.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T03:59:59.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T03:59:59.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:00.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:00.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:00.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:00.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:00.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:00:00.252+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:00.257434+0000) 2019-10-18T04:00:01.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:01.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:01.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:01.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:02.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:02.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:02.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:02.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:02.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:02.092+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:03.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:03.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:03.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:03.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:03.882 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:00:03.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:03.883160+0000) 2019-10-18T04:00:04.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:04.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:04.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:04.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:05.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:05.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:05.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:05.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:05.393 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:00:05.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:05.394763+0000) 2019-10-18T04:00:05.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:00:05.720+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:05.725347+0000) 2019-10-18T04:00:06.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:06.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:06.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:06.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:06.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:06.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:06.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:06.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:07.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:07.092+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:07.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:07.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:07.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:07.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:08.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:08.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:08.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:08.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:09.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:09.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:09.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:09.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:10.256 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:00:10.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:10.257557+0000) 2019-10-18T04:00:10.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:10.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:10.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:10.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:11.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:11.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:11.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:11.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:12.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:12.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:12.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:12.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:12.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:12.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:13.882 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:00:13.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:13.883346+0000) 2019-10-18T04:00:13.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:13.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:13.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:13.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:14.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:14.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:14.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:14.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:15.394 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:00:15.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:15.394972+0000) 2019-10-18T04:00:15.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:00:15.723+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:15.725456+0000) 2019-10-18T04:00:15.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:15.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:15.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:15.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:16.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:16.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:16.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:16.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:17.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:17.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:17.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:17.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:17.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:17.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:18.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:18.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:18.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:18.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:20.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:20.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:20.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:20.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:20.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:00:20.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:20.257655+0000) 2019-10-18T04:00:20.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:20.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:20.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:20.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:21.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:21.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:21.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:21.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:22.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:22.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:22.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:22.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:22.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:22.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:23.882 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:00:23.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:23.883532+0000) 2019-10-18T04:00:23.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:23.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:23.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:23.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:24.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:24.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:24.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:24.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:25.394 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:00:25.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:25.395113+0000) 2019-10-18T04:00:25.724 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:00:25.723+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:25.725555+0000) 2019-10-18T04:00:25.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:25.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:25.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:25.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:26.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:26.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:26.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:26.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:27.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:27.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:27.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:27.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:27.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:27.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:28.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:28.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:28.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:28.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:29.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:29.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:29.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:29.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:30.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:00:30.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:30.257753+0000) 2019-10-18T04:00:30.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:30.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:30.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:30.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:31.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:31.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:31.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:31.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:32.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:32.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:33.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:33.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:33.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:33.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:33.882 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:00:33.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:33.883725+0000) 2019-10-18T04:00:34.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:34.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:34.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:34.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:35.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:35.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:35.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:35.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:35.394 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:00:35.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:35.395295+0000) 2019-10-18T04:00:35.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:00:35.723+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:35.725655+0000) 2019-10-18T04:00:36.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:36.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:36.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:36.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:37.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:37.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:37.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:37.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:37.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:37.095+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:38.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:38.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:38.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:38.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:39.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:39.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:39.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:39.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:40.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:40.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:40.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:40.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:40.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:00:40.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:40.257854+0000) 2019-10-18T04:00:41.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:41.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:41.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:41.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:42.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:42.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:42.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:42.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:42.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:42.099+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:43.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:43.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:43.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:43.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:43.883 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:00:43.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:43.883941+0000) 2019-10-18T04:00:44.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:44.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:44.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:44.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:45.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:45.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:45.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:45.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:45.394 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:00:45.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:45.395482+0000) 2019-10-18T04:00:45.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:00:45.722+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:45.725749+0000) 2019-10-18T04:00:46.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:46.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:46.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:46.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:47.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:47.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:47.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:47.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:47.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:47.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:48.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:48.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:48.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:48.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:49.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:49.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:49.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:49.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:50.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:50.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:50.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:50.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:50.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:00:50.254+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:50.257954+0000) 2019-10-18T04:00:51.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:51.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:51.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:51.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:52.103 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:52.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:52.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:52.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:52.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:52.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:53.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:53.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:53.883 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:00:53.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:53.884122+0000) 2019-10-18T04:00:54.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:54.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:54.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:54.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:55.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:55.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:55.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:55.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:55.394 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:00:55.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:55.395666+0000) 2019-10-18T04:00:55.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:00:55.722+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:00:55.725845+0000) 2019-10-18T04:00:56.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:56.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:56.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:56.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:57.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:00:57.098+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:00:57.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:57.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:57.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:57.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:58.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:58.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:58.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:58.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:00:59.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:59.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:00:59.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:00:59.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:00.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:00.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:00.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:00.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:00.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:01:00.254+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:00.258052+0000) 2019-10-18T04:01:01.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:01.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:01.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:01.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:02.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:02.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:02.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:02.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:02.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:02.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:03.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:03.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:03.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:03.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:03.883 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:01:03.881+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:03.884306+0000) 2019-10-18T04:01:04.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:04.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:04.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:04.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:05.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:05.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:05.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:05.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:05.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:01:05.393+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:05.395833+0000) 2019-10-18T04:01:05.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:01:05.722+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:05.725944+0000) 2019-10-18T04:01:06.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:06.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:06.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:06.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:07.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:07.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:07.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:07.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:07.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:07.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:08.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:08.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:08.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:08.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:09.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:09.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:09.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:09.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:10.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:01:10.254+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:10.258150+0000) 2019-10-18T04:01:10.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:10.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:10.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:10.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:11.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:11.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:11.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:11.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:12.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:12.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:12.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:12.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:12.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:12.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:13.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:13.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:13.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:13.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:13.883 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:01:13.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:13.884490+0000) 2019-10-18T04:01:14.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:14.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:14.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:14.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:15.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:15.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:15.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:15.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:15.396 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:01:15.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:15.395963+0000) 2019-10-18T04:01:15.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:01:15.722+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:15.726039+0000) 2019-10-18T04:01:16.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:16.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:16.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:16.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:17.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:17.102+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:17.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:17.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:17.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:17.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:18.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:18.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:18.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:18.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:19.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:19.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:19.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:19.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:20.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:01:20.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:20.258255+0000) 2019-10-18T04:01:20.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:20.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:20.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:20.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:21.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:21.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:21.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:21.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:22.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:22.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:22.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:22.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:22.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:22.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:23.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:23.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:23.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:23.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:23.883 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:01:23.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:23.884671+0000) 2019-10-18T04:01:24.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:24.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:24.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:24.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:25.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:01:25.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:25.396112+0000) 2019-10-18T04:01:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:25.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:25.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:25.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:01:25.721+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:25.726145+0000) 2019-10-18T04:01:26.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:26.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:26.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:26.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:27.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:27.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:27.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:27.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:27.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:27.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:28.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:28.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:28.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:28.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:29.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:29.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:29.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:29.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:30.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:01:30.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:30.258355+0000) 2019-10-18T04:01:30.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:30.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:30.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:30.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:31.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:31.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:31.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:31.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:32.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:32.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:32.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:32.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:32.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:32.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:33.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:33.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:33.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:33.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:33.883 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:01:33.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:33.884856+0000) 2019-10-18T04:01:34.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:34.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:34.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:34.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:35.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:01:35.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:35.396281+0000) 2019-10-18T04:01:35.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:35.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:35.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:35.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:35.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:01:35.721+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:35.726247+0000) 2019-10-18T04:01:36.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:36.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:36.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:36.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:37.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:37.105+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:37.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:37.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:37.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:37.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:38.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:38.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:38.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:38.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:39.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:39.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:39.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:39.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:40.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:01:40.257+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:40.258461+0000) 2019-10-18T04:01:40.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:40.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:40.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:40.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:41.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:41.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:41.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:41.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:42.111 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:42.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:42.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:42.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:42.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:42.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:43.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:43.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:43.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:43.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:43.884 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:01:43.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:43.885039+0000) 2019-10-18T04:01:44.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:44.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:44.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:44.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:45.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:45.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:45.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:45.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:45.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:01:45.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:45.396451+0000) 2019-10-18T04:01:45.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:01:45.725+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:45.726347+0000) 2019-10-18T04:01:46.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:46.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:46.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:46.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:47.112 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:47.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:47.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:47.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:47.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:47.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:48.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:48.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:48.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:48.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:49.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:49.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:49.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:49.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:50.257 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:01:50.257+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:50.258562+0000) 2019-10-18T04:01:50.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:50.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:50.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:50.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:51.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:51.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:51.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:51.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:52.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:52.113+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:52.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:52.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:52.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:52.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:53.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:53.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:53.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:53.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:53.884 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:01:53.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:53.885224+0000) 2019-10-18T04:01:54.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:54.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:54.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:54.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:55.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:01:55.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:55.396606+0000) 2019-10-18T04:01:55.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:55.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:55.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:55.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:55.725 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:01:55.724+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:01:55.726445+0000) 2019-10-18T04:01:56.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:56.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:56.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:56.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:57.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:01:57.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:01:57.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:57.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:57.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:57.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:58.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:58.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:58.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:58.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:01:59.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:59.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:01:59.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:01:59.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:00.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:02:00.256+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:00.258661+0000) 2019-10-18T04:02:00.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:00.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:00.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:00.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:01.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:01.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:01.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:01.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:02.116 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:02.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:02.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:02.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:02.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:02.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:03.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:03.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:03.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:03.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:03.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:02:03.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:03.885437+0000) 2019-10-18T04:02:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:04.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:04.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:05.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:02:05.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:05.396754+0000) 2019-10-18T04:02:05.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:05.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:05.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:05.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:05.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:02:05.724+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:05.726543+0000) 2019-10-18T04:02:06.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:06.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:06.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:06.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:07.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:07.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:07.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:07.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:07.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:07.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:08.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:08.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:08.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:08.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:09.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:09.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:09.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:09.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:10.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:02:10.256+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:10.258757+0000) 2019-10-18T04:02:10.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:10.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:10.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:10.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:11.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:11.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:11.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:11.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:12.116 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:12.112+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:12.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:12.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:12.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:12.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:13.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:13.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:13.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:13.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:13.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:02:13.880+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:13.885623+0000) 2019-10-18T04:02:14.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:14.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:14.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:14.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:15.395 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:02:15.392+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:15.396890+0000) 2019-10-18T04:02:15.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:15.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:15.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:15.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:15.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:02:15.724+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:15.726655+0000) 2019-10-18T04:02:16.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:16.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:16.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:16.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:17.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:17.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:17.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:17.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:17.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:17.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:18.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:18.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:18.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:18.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:19.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:19.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:19.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:19.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:20.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:02:20.256+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:20.258852+0000) 2019-10-18T04:02:20.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:20.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:20.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:20.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:21.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:21.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:21.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:21.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:22.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:22.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:22.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:22.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:22.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:22.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:23.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:23.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:23.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:23.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:23.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:02:23.879+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:23.885805+0000) 2019-10-18T04:02:24.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:24.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:24.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:24.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:25.396 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:02:25.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:25.397061+0000) 2019-10-18T04:02:25.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:25.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:25.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:25.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:25.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:02:25.724+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:25.726743+0000) 2019-10-18T04:02:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:26.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:26.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:27.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:27.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:27.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:27.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:27.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:27.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:28.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:28.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:28.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:28.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:29.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:29.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:29.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:29.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:30.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:02:30.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:30.258951+0000) 2019-10-18T04:02:30.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:30.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:30.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:30.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:31.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:31.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:31.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:31.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:32.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:32.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:32.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:32.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:32.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:32.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:33.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:33.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:33.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:33.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:33.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:02:33.879+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:33.885987+0000) 2019-10-18T04:02:34.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:34.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:34.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:34.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:35.396 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:02:35.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:35.397211+0000) 2019-10-18T04:02:35.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:35.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:35.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:35.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:35.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:02:35.723+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:35.726843+0000) 2019-10-18T04:02:36.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:36.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:36.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:36.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:37.120 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:37.115+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:37.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:37.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:37.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:37.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:38.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:38.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:38.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:38.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:39.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:39.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:39.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:39.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:40.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:02:40.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:40.259055+0000) 2019-10-18T04:02:40.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:40.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:40.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:40.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:41.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:41.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:41.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:41.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:42.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:42.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:42.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:42.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:42.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:42.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:43.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:43.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:43.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:43.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:43.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:02:43.883+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:43.886199+0000) 2019-10-18T04:02:44.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:44.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:44.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:44.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:45.396 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:02:45.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:45.397399+0000) 2019-10-18T04:02:45.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:45.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:45.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:45.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:45.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:02:45.723+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:45.726943+0000) 2019-10-18T04:02:46.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:46.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:46.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:46.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:47.122 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:47.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:47.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:47.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:47.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:47.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:48.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:48.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:48.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:48.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:49.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:49.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:49.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:49.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:50.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:02:50.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:50.259156+0000) 2019-10-18T04:02:50.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:50.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:50.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:50.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:51.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:51.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:51.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:51.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:52.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:52.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:52.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:52.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:52.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:52.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:53.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:53.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:53.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:53.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:53.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:02:53.883+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:53.886388+0000) 2019-10-18T04:02:54.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:54.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:54.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:54.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:55.397 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:02:55.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:55.397586+0000) 2019-10-18T04:02:55.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:55.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:55.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:55.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:55.727 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:02:55.723+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:02:55.727041+0000) 2019-10-18T04:02:56.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:56.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:56.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:56.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:57.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:02:57.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:02:57.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:57.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:57.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:57.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:58.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:58.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:58.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:58.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:02:59.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:59.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:02:59.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:02:59.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:00.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:03:00.255+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:00.259254+0000) 2019-10-18T04:03:00.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:00.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:00.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:00.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:01.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:01.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:01.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:01.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:02.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:02.119+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:02.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:02.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:02.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:02.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:03.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:03.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:03.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:03:03.883+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:03.886533+0000) 2019-10-18T04:03:04.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:04.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:04.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:04.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:05.396 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:03:05.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:05.397764+0000) 2019-10-18T04:03:05.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:05.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:05.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:05.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:05.726 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:05.722+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:05.727143+0000) 2019-10-18T04:03:06.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:06.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:06.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:06.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:07.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:07.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:07.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:07.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:07.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:07.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:08.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:08.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:08.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:08.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:09.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:09.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:09.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:09.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:10.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:03:10.254+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:10.259354+0000) 2019-10-18T04:03:10.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:10.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:10.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:10.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:11.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:11.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:11.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:11.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:12.126 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:12.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:12.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:12.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:12.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:12.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:13.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:13.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:13.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:13.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:13.885 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:03:13.883+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:13.886713+0000) 2019-10-18T04:03:14.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:14.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:14.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:14.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:15.397 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:03:15.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:15.397953+0000) 2019-10-18T04:03:15.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:15.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:15.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:15.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:15.727 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:15.722+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:15.727237+0000) 2019-10-18T04:03:15.727 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:15.722+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:15.728249+0000) 2019-10-18T04:03:16.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:16.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:16.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:16.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:17.127 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:17.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:17.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:17.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:17.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:17.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:18.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:18.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:18.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:18.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:19.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:19.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:19.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:19.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:20.226 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:20.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:20.227430+0000) 2019-10-18T04:03:20.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:03:20.254+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:20.259453+0000) 2019-10-18T04:03:20.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:20.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:20.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:20.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:21.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:21.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:22.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:22.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:22.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:22.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:22.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:22.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:23.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:23.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:23.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:23.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:23.886 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:03:23.883+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:23.886894+0000) 2019-10-18T04:03:24.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:24.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:24.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:24.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:25.397 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:03:25.391+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:25.398137+0000) 2019-10-18T04:03:25.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:25.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:25.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:25.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:26.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:26.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:26.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:26.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:27.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:27.122+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:27.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:27.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:27.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:27.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:28.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:28.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:28.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:28.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:29.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:29.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:29.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:29.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:30.226 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:30.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:30.227531+0000) 2019-10-18T04:03:30.258 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:03:30.254+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:30.259549+0000) 2019-10-18T04:03:30.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:30.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:30.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:30.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:31.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:31.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:31.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:31.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:32.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:32.126+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:32.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:32.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:32.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:32.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:33.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:33.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:33.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:33.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:33.886 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:03:33.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:33.887073+0000) 2019-10-18T04:03:34.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:34.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:34.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:34.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:35.397 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:03:35.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:35.398356+0000) 2019-10-18T04:03:35.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:35.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:35.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:35.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:36.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:36.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:36.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:36.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:37.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:37.126+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:37.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:37.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:37.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:37.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:38.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:38.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:38.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:38.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:39.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:39.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:39.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:39.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:40.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:40.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:40.227630+0000) 2019-10-18T04:03:40.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:03:40.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:40.259649+0000) 2019-10-18T04:03:40.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:40.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:40.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:40.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:41.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:41.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:41.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:41.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:42.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:42.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:42.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:42.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:42.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:42.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:43.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:43.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:43.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:43.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:43.886 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:03:43.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:43.887289+0000) 2019-10-18T04:03:44.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:44.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:44.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:44.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:45.400 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:03:45.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:45.398499+0000) 2019-10-18T04:03:45.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:45.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:45.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:45.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:46.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:46.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:46.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:46.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:47.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:47.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:47.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:47.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:47.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:47.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:48.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:48.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:48.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:48.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:49.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:49.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:49.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:49.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:50.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:03:50.221+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:50.227730+0000) 2019-10-18T04:03:50.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:03:50.253+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:50.259746+0000) 2019-10-18T04:03:50.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:50.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:50.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:50.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:51.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:51.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:51.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:51.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:52.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:52.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:52.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:52.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:52.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:52.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:53.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:53.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:53.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:53.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:53.887 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:03:53.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:53.887428+0000) 2019-10-18T04:03:54.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:54.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:54.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:54.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:55.397 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:03:55.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:03:55.398669+0000) 2019-10-18T04:03:55.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:55.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:55.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:55.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:56.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:56.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:56.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:56.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:57.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:03:57.129+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:03:57.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:57.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:57.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:57.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:58.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:58.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:58.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:58.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:03:59.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:59.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:03:59.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:03:59.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:00.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:04:00.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:00.227835+0000) 2019-10-18T04:04:00.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:00.257+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:00.259839+0000) 2019-10-18T04:04:00.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:00.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:04:00.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:00.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:01.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:01.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T03:53:30.574355+0000 (oldest deadline 2019-10-18T03:53:50.574355+0000) 2019-10-18T04:04:01.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:01.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:02.135 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:02.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:02.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:02.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:03.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:03.886 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:04:03.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:03.887619+0000) 2019-10-18T04:04:04.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:04.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:05.397 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:04:05.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:05.398849+0000) 2019-10-18T04:04:05.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:05.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:06.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:06.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:07.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:07.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:07.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:07.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:08.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:08.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:09.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:09.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:10.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:04:10.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:10.227934+0000) 2019-10-18T04:04:10.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:10.257+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:10.259931+0000) 2019-10-18T04:04:10.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:10.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:11.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:11.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:12.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:12.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:12.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:12.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:13.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:13.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:13.892 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:04:13.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:13.887809+0000) 2019-10-18T04:04:14.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:14.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:15.398 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:04:15.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:15.399026+0000) 2019-10-18T04:04:15.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:15.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:16.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:16.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:17.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:17.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:17.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:17.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:18.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:18.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:19.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:19.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:20.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:04:20.224+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:20.228030+0000) 2019-10-18T04:04:20.259 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:20.256+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:20.260030+0000) 2019-10-18T04:04:20.260 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:20.256+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:20.261178+0000) 2019-10-18T04:04:20.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:20.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:21.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:21.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:22.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:22.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:22.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:22.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:23.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:23.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:23.887 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:04:23.882+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:23.888002+0000) 2019-10-18T04:04:24.759 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:24.756+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:24.760222+0000) 2019-10-18T04:04:24.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:24.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:25.398 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:04:25.394+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:25.399207+0000) 2019-10-18T04:04:25.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:25.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:25.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:25.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:26.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:26.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:26.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:26.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:27.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:27.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:27.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:27.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:27.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:27.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:28.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:28.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:28.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:28.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:29.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:29.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:29.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:29.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:30.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:04:30.224+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:30.228122+0000) 2019-10-18T04:04:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:30.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:30.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:31.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:31.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:31.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:31.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:32.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:32.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:32.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:32.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:32.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:32.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:33.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:33.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:33.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:33.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:33.888 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:04:33.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:33.888181+0000) 2019-10-18T04:04:34.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:34.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:34.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:34.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:34.759 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:34.756+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:34.760317+0000) 2019-10-18T04:04:35.398 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:04:35.398+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:35.399389+0000) 2019-10-18T04:04:35.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:35.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:35.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:35.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:36.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:36.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:36.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:36.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:37.140 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:37.136+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:37.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:37.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:37.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:37.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:38.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:38.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:38.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:38.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:39.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:39.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:39.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:39.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:40.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:04:40.224+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:40.228218+0000) 2019-10-18T04:04:40.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:40.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:40.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:40.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:41.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:41.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:41.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:41.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:42.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:42.140+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:42.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:42.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:42.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:42.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:43.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:43.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:43.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:43.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:43.887 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:04:43.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:43.888369+0000) 2019-10-18T04:04:44.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:44.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:44.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:44.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:44.759 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:44.756+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:44.760416+0000) 2019-10-18T04:04:45.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:45.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:45.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:45.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:45.398 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:04:45.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:45.399566+0000) 2019-10-18T04:04:46.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:46.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:46.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:46.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:47.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:47.140+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:47.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:47.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:47.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:47.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:48.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:48.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:48.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:48.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:49.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:49.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:49.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:49.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:50.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:04:50.224+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:50.228317+0000) 2019-10-18T04:04:50.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:50.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:50.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:50.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:51.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:51.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:51.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:51.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:52.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:52.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:52.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:52.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:52.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:52.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:53.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:53.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:53.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:53.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:53.887 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:04:53.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:53.888552+0000) 2019-10-18T04:04:54.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:54.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:54.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:54.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:54.759 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:04:54.755+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:54.760515+0000) 2019-10-18T04:04:55.398 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:04:55.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:04:55.399743+0000) 2019-10-18T04:04:55.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:55.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:55.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:55.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:56.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:56.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:56.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:56.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:57.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:04:57.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:04:57.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:57.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:57.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:57.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:58.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:58.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:58.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:58.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:04:59.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:59.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:04:59.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:04:59.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:00.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:05:00.223+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:00.228425+0000) 2019-10-18T04:05:00.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:00.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:00.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:00.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:01.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:01.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:01.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:01.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:02.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:02.139+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:02.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:02.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:02.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:02.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:03.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:03.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:03.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:03.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:03.887 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:05:03.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:03.888732+0000) 2019-10-18T04:05:04.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:04.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:04.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:04.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:04.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:05:04.755+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:04.760610+0000) 2019-10-18T04:05:05.398 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:05.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:05.399923+0000) 2019-10-18T04:05:05.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:05.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:05.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:05.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:06.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:06.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:06.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:06.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:07.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:07.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:07.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:07.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:07.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:07.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:08.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:08.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:08.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:08.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:09.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:09.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:09.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:09.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:10.227 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:05:10.223+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:10.228522+0000) 2019-10-18T04:05:10.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:10.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:10.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:10.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:11.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:11.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:11.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:11.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:12.146 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:12.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:12.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:12.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:12.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:12.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:13.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:13.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:13.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:13.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:13.888 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:05:13.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:13.888921+0000) 2019-10-18T04:05:14.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:14.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:14.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:14.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:14.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:05:14.755+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:14.760711+0000) 2019-10-18T04:05:15.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:15.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:15.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:15.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:15.399 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:15.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:15.400105+0000) 2019-10-18T04:05:16.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:16.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:16.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:16.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:17.147 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:17.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:17.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:17.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:17.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:17.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:18.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:18.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:18.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:18.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:19.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:19.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:19.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:19.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:20.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:05:20.223+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:20.228622+0000) 2019-10-18T04:05:20.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:20.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:20.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:20.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:21.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:21.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:21.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:21.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:22.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:22.143+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:22.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:22.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:23.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:23.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:23.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:23.888 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:05:23.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:23.889148+0000) 2019-10-18T04:05:24.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:24.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:24.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:24.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:24.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:05:24.755+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:24.760810+0000) 2019-10-18T04:05:25.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:25.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:25.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:25.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:25.400 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:25.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:25.400285+0000) 2019-10-18T04:05:26.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:26.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:26.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:27.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:27.146+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:27.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:27.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:27.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:27.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:28.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:28.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:28.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:28.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:29.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:29.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:29.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:29.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:30.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:05:30.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:30.228723+0000) 2019-10-18T04:05:30.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:30.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:30.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:30.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:31.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:31.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:31.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:31.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:32.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:32.146+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:32.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:32.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:32.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:32.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:33.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:33.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:33.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:33.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:33.888 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:05:33.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:33.889333+0000) 2019-10-18T04:05:34.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:34.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:34.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:34.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:34.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:05:34.754+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:34.760905+0000) 2019-10-18T04:05:35.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:35.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:35.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:35.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:35.399 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:35.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:35.400470+0000) 2019-10-18T04:05:36.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:36.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:36.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:36.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:37.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:37.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:37.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:37.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:37.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:37.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:38.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:38.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:38.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:38.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:39.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:39.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:39.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:39.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:40.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:40.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:40.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:40.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:40.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:05:40.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:40.228817+0000) 2019-10-18T04:05:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:41.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:41.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:41.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:42.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:42.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:42.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:42.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:42.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:42.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:43.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:43.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:43.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:43.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:43.888 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:05:43.885+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:43.889513+0000) 2019-10-18T04:05:44.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:44.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:44.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:44.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:44.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:05:44.754+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:44.761001+0000) 2019-10-18T04:05:45.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:45.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:45.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:45.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:45.399 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:45.397+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:45.400642+0000) 2019-10-18T04:05:46.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:46.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:46.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:46.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:47.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:47.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:47.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:47.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:47.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:47.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:48.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:48.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:48.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:48.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:49.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:49.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:49.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:49.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:50.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:50.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:50.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:50.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:50.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:05:50.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:50.228916+0000) 2019-10-18T04:05:51.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:51.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:51.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:51.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:52.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:52.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:52.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:52.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:52.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:52.150+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:53.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:53.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:53.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:53.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:53.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:05:53.884+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:53.889702+0000) 2019-10-18T04:05:54.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:54.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:54.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:54.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:54.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:05:54.754+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:54.761101+0000) 2019-10-18T04:05:55.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:55.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:55.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:55.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:55.399 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:55.396+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:55.400814+0000) 2019-10-18T04:05:55.400 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:55.396+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:55.401902+0000) 2019-10-18T04:05:56.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:56.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:56.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:56.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:57.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:57.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:57.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:57.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:57.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:05:57.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:05:58.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:58.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:58.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:58.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:59.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:59.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:05:59.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:05:59.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:05:59.900 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:05:59.896+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:05:59.901405+0000) 2019-10-18T04:06:00.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:00.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:00.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:00.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:00.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:06:00.222+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:00.228992+0000) 2019-10-18T04:06:01.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:01.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:01.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:01.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:02.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:02.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:02.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:02.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:02.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:02.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:03.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:03.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:03.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:03.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:03.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:06:03.884+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:03.889889+0000) 2019-10-18T04:06:04.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:04.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:04.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:04.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:04.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:06:04.753+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:04.761198+0000) 2019-10-18T04:06:05.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:05.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:05.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:05.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:06.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:06.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:06.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:06.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:07.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:07.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:07.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:07.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:07.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:07.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:08.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:08.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:08.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:08.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:09.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:09.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:09.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:09.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:09.900 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:06:09.896+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:09.901616+0000) 2019-10-18T04:06:10.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:10.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:10.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:10.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:10.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:06:10.221+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:10.229089+0000) 2019-10-18T04:06:11.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:11.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:11.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:11.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:12.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:12.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:12.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:12.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:12.159 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:12.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:13.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:13.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:13.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:13.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:13.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:06:13.884+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:13.890077+0000) 2019-10-18T04:06:14.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:14.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:14.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:14.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:14.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:06:14.753+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:14.761299+0000) 2019-10-18T04:06:15.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:15.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:15.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:15.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:16.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:16.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:16.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:16.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:17.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:17.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:17.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:17.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:17.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:17.153+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:18.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:18.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:18.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:18.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:19.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:19.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:19.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:19.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:19.901 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:06:19.896+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:19.901799+0000) 2019-10-18T04:06:20.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:20.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:20.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:20.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:20.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:06:20.221+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:20.229186+0000) 2019-10-18T04:06:21.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:21.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:21.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:21.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:22.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:22.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:22.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:22.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:22.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:22.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:23.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:23.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:23.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:23.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:23.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:06:23.888+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:23.890266+0000) 2019-10-18T04:06:24.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:24.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:24.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:24.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:24.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:06:24.757+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:24.761393+0000) 2019-10-18T04:06:25.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:25.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:25.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:25.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:26.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:26.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:26.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:26.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:27.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:27.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:27.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:27.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:27.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:27.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:28.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:28.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:28.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:28.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:29.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:29.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:29.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:29.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:29.901 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:06:29.896+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:29.901984+0000) 2019-10-18T04:06:30.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:06:30.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:30.229284+0000) 2019-10-18T04:06:30.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:30.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:30.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:30.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:31.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:31.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:31.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:31.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:32.164 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:32.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:32.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:32.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:32.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:32.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:33.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:33.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:33.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:33.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:33.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:06:33.888+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:33.890447+0000) 2019-10-18T04:06:34.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:34.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:34.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:34.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:34.760 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:06:34.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:34.761498+0000) 2019-10-18T04:06:35.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:35.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:35.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:35.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:36.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:36.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:36.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:36.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:37.164 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:37.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:37.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:37.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:37.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:37.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:38.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:38.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:38.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:38.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:39.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:39.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:39.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:39.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:39.901 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:06:39.900+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:39.902164+0000) 2019-10-18T04:06:40.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:06:40.224+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:40.229381+0000) 2019-10-18T04:06:40.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:40.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:40.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:40.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:41.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:41.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:41.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:41.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:42.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:42.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:42.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:42.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:42.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:42.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:43.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:43.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:43.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:43.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:43.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:06:43.888+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:43.890627+0000) 2019-10-18T04:06:44.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:44.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:44.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:44.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:44.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:06:44.760+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:44.761603+0000) 2019-10-18T04:06:45.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:45.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:45.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:45.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:46.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:46.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:46.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:46.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:47.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:47.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:47.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:47.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:47.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:47.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:48.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:48.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:48.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:48.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:49.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:49.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:49.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:49.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:49.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:06:49.900+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:49.902383+0000) 2019-10-18T04:06:50.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:06:50.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:50.229470+0000) 2019-10-18T04:06:50.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:50.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:50.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:50.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:51.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:51.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:51.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:51.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:52.167 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:52.164+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:52.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:52.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:52.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:52.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:53.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:53.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:53.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:53.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:53.889 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:06:53.888+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:53.890797+0000) 2019-10-18T04:06:54.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:54.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:54.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:54.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:54.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:06:54.760+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:54.761701+0000) 2019-10-18T04:06:55.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:55.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:55.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:55.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:56.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:56.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:56.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:56.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:57.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:06:57.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:06:57.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:57.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:57.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:57.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:58.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:58.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:58.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:58.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:59.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:59.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:06:59.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:06:59.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:06:59.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:06:59.900+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:06:59.902563+0000) 2019-10-18T04:07:00.228 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:07:00.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:00.229570+0000) 2019-10-18T04:07:00.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:00.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:00.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:00.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:01.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:01.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:01.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:01.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:02.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:02.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:02.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:02.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:02.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:02.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:03.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:03.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:03.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:03.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:03.890 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:07:03.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:03.890977+0000) 2019-10-18T04:07:04.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:04.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:04.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:04.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:04.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:07:04.760+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:04.761800+0000) 2019-10-18T04:07:05.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:05.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:05.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:05.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:06.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:06.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:06.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:06.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:07.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:07.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:07.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:07.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:07.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:07.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:08.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:08.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:08.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:08.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:09.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:09.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:09.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:09.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:09.901 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:07:09.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:09.902750+0000) 2019-10-18T04:07:10.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:07:10.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:10.229676+0000) 2019-10-18T04:07:10.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:10.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:10.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:10.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:11.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:11.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:11.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:11.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:12.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:12.167+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:12.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:12.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:12.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:12.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:13.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:13.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:13.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:13.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:13.891 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:07:13.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:13.891168+0000) 2019-10-18T04:07:14.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:14.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:14.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:14.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:14.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:07:14.759+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:14.761900+0000) 2019-10-18T04:07:15.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:15.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:15.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:15.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:16.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:16.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:16.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:16.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:17.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:17.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:17.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:17.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:17.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:17.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:18.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:18.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:18.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:18.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:19.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:19.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:19.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:19.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:19.902 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:07:19.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:19.902930+0000) 2019-10-18T04:07:20.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:20.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:20.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:20.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:20.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:07:20.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:20.229773+0000) 2019-10-18T04:07:21.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:21.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:21.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:21.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:22.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:22.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:22.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:22.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:22.176 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:22.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:23.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:23.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:23.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:23.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:23.890 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:07:23.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:23.891353+0000) 2019-10-18T04:07:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:24.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:24.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:24.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:24.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:07:24.759+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:24.761998+0000) 2019-10-18T04:07:25.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:25.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:25.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:25.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:26.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:26.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:26.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:26.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:27.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:27.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:27.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:27.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:27.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:27.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:28.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:28.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:28.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:28.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:29.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:29.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:29.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:29.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:29.902 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:07:29.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:29.903117+0000) 2019-10-18T04:07:30.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:30.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:30.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:30.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:30.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:07:30.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:30.229876+0000) 2019-10-18T04:07:31.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:31.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:31.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:31.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:32.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:32.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:32.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:32.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:32.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:32.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:33.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:33.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:33.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:33.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:33.892 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:07:33.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:33.891543+0000) 2019-10-18T04:07:34.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:34.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:34.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:34.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:34.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:07:34.759+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:34.762091+0000) 2019-10-18T04:07:35.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:35.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:35.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:35.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:36.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:36.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:36.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:36.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:37.179 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:37.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:37.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:37.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:37.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:37.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:38.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:38.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:38.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:38.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:39.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:39.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:39.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:39.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:39.902 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:07:39.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:39.903299+0000) 2019-10-18T04:07:40.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:40.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:40.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:40.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:40.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:07:40.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:40.229975+0000) 2019-10-18T04:07:41.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:41.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:41.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:41.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:42.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:42.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:42.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:42.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:42.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:42.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:43.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:43.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:43.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:43.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:43.890 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:07:43.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:43.891736+0000) 2019-10-18T04:07:44.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:44.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:44.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:44.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:44.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:07:44.759+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:44.762187+0000) 2019-10-18T04:07:45.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:45.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:45.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:45.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:46.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:46.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:46.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:46.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:47.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:47.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:47.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:47.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:47.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:47.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:48.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:48.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:48.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:48.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:49.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:49.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:49.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:49.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:49.902 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:07:49.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:49.903489+0000) 2019-10-18T04:07:50.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:50.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:50.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:50.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:50.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:07:50.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:50.230075+0000) 2019-10-18T04:07:51.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:51.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:51.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:51.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:52.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:52.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:52.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:52.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:52.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:52.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:53.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:53.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:53.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:53.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:53.891 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:07:53.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:53.891927+0000) 2019-10-18T04:07:54.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:54.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:54.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:54.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:54.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:07:54.758+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:54.762287+0000) 2019-10-18T04:07:55.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:55.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:55.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:55.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:56.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:56.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:56.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:56.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:57.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:07:57.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:07:57.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:57.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:57.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:57.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:58.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:58.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:58.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:58.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:59.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:59.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:07:59.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:07:59.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:07:59.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:07:59.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:07:59.903671+0000) 2019-10-18T04:08:00.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:00.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:00.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:00.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:00.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:08:00.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:00.230188+0000) 2019-10-18T04:08:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:01.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:01.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:02.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:02.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:02.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:02.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:02.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:02.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:03.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:03.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:03.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:03.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:03.891 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:08:03.887+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:03.892108+0000) 2019-10-18T04:08:04.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:04.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:04.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:04.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:04.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:08:04.758+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:04.762382+0000) 2019-10-18T04:08:05.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:05.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:05.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:05.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:06.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:06.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:06.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:06.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:07.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:07.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:07.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:07.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:07.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:07.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:08.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:08.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:08.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:08.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:09.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:09.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:09.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:09.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:09.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:08:09.899+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:09.903855+0000) 2019-10-18T04:08:10.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:10.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:10.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:10.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:10.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:08:10.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:10.230286+0000) 2019-10-18T04:08:11.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:11.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:11.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:11.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:12.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:12.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:12.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:12.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:12.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:12.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:13.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:13.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:13.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:13.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:13.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:08:13.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:13.892249+0000) 2019-10-18T04:08:14.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:14.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:14.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:14.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:14.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:08:14.758+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:14.762481+0000) 2019-10-18T04:08:15.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:15.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:15.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:15.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:16.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:16.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:16.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:16.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:17.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:17.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:17.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:17.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:17.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:17.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:18.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:18.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:19.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:19.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:19.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:19.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:19.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:08:19.898+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:19.904039+0000) 2019-10-18T04:08:20.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:20.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:20.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:20.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:20.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:08:20.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:20.230342+0000) 2019-10-18T04:08:21.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:21.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:21.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:21.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:22.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:22.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:22.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:22.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:22.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:22.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:23.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:23.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:23.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:23.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:23.891 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:08:23.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:23.892451+0000) 2019-10-18T04:08:24.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:24.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:24.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:24.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:24.761 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:08:24.757+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:24.762584+0000) 2019-10-18T04:08:25.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:25.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:25.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:25.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:26.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:26.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:26.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:26.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:26.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:26.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:26.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:26.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:27.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:27.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:27.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:27.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:27.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:27.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:28.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:28.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:28.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:28.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:29.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:08:29.898+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:29.904219+0000) 2019-10-18T04:08:29.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:29.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:29.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:29.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:30.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:08:30.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:30.230443+0000) 2019-10-18T04:08:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:30.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:30.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:31.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:31.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:31.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:31.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:32.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:32.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:32.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:32.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:32.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:32.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:33.891 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:08:33.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:33.892625+0000) 2019-10-18T04:08:33.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:33.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:33.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:33.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:34.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:08:34.757+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:34.762677+0000) 2019-10-18T04:08:34.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:34.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:34.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:34.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:35.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:35.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:35.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:35.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:37.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:37.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:37.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:37.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:37.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:37.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:37.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:37.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:37.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:37.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:39.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:39.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:39.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:39.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:39.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:08:39.898+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:39.904402+0000) 2019-10-18T04:08:39.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:39.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:39.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:39.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:40.229 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:08:40.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:40.230542+0000) 2019-10-18T04:08:40.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:40.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:40.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:40.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:42.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:41.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:42.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:42.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:42.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:42.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:42.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:42.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:42.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:42.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:43.891 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:08:43.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:43.892814+0000) 2019-10-18T04:08:43.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:43.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:43.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:43.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:44.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:08:44.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:44.762776+0000) 2019-10-18T04:08:45.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:45.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:45.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:45.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:46.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:46.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:46.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:46.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:46.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:46.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:46.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:46.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:47.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:47.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:47.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:47.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:47.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:47.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:48.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:48.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:48.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:48.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:49.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:08:49.898+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:49.904598+0000) 2019-10-18T04:08:49.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:49.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:49.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:49.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:50.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:08:50.229+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:50.230644+0000) 2019-10-18T04:08:50.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:50.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:50.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:50.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:51.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:51.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:51.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:51.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:52.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:52.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:52.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:52.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:52.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:52.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:53.892 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:08:53.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:53.893006+0000) 2019-10-18T04:08:53.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:53.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:53.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:53.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:54.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:08:54.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:54.762877+0000) 2019-10-18T04:08:54.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:54.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:54.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:54.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:55.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:55.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:55.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:55.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:57.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:57.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:57.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:57.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:57.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:08:57.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:08:58.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:58.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:58.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:58.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:59.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:59.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:08:59.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:08:59.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:08:59.903 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:08:59.898+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:08:59.904782+0000) 2019-10-18T04:09:00.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:00.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:00.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:00.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:00.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:09:00.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:00.230750+0000) 2019-10-18T04:09:01.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:01.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:01.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:01.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:02.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:02.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:02.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:02.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:02.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:02.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:03.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:03.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:03.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:03.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:03.892 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:09:03.886+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:03.893187+0000) 2019-10-18T04:09:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:04.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:04.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:04.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:04.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:09:04.760+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:04.762972+0000) 2019-10-18T04:09:05.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:05.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:05.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:05.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:06.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:06.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:06.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:06.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:07.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:07.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:07.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:07.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:07.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:08.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:08.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:08.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:08.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:09.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:09.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:09.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:09.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:09.904 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:09:09.898+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:09.904968+0000) 2019-10-18T04:09:10.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:10.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:10.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:10.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:10.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:09:10.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:10.230843+0000) 2019-10-18T04:09:10.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:10.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:10.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:10.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:11.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:11.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:11.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:11.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:12.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:12.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:12.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:12.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:12.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:12.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:13.892 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:09:13.890+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:13.893375+0000) 2019-10-18T04:09:14.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:13.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:14.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:14.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:14.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:09:14.760+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:14.763071+0000) 2019-10-18T04:09:15.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:15.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:15.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:15.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:15.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:15.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:15.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:15.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:16.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:16.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:16.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:16.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:17.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:17.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:17.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:17.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:17.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:17.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:18.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:18.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:18.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:18.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:19.904 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:09:19.902+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:19.905151+0000) 2019-10-18T04:09:19.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:19.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:19.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:19.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:20.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:09:20.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:20.230941+0000) 2019-10-18T04:09:21.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:21.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:21.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:21.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:21.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:21.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:21.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:21.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:22.202 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:22.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:22.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:22.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:22.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:22.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:23.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:09:23.889+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:23.893564+0000) 2019-10-18T04:09:23.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:23.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:23.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:23.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:24.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:09:24.760+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:24.763172+0000) 2019-10-18T04:09:24.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:24.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:24.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:24.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:26.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:26.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:26.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:26.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:27.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:27.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:27.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:27.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:27.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:27.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:28.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:28.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:28.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:28.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:29.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:29.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:29.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:29.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:29.904 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:09:29.901+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:29.905307+0000) 2019-10-18T04:09:30.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:30.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:30.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:30.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:30.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:09:30.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:30.231042+0000) 2019-10-18T04:09:31.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:31.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:31.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:31.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:32.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:32.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:32.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:32.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:32.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:32.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:33.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:33.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:33.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:33.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:33.892 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:09:33.889+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:33.893754+0000) 2019-10-18T04:09:34.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:34.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:34.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:34.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:34.762 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:09:34.759+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:34.763266+0000) 2019-10-18T04:09:35.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:35.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:35.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:35.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:36.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:36.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:36.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:36.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:37.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:37.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:37.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:37.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:37.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:37.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:38.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:38.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:38.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:38.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:39.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:39.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:39.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:39.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:39.904 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:09:39.901+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:39.905520+0000) 2019-10-18T04:09:40.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:40.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:40.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:40.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:40.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:09:40.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:40.231138+0000) 2019-10-18T04:09:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:41.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:41.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:42.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:42.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:42.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:42.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:42.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:42.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:43.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:43.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:43.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:43.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:09:43.889+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:43.893948+0000) 2019-10-18T04:09:44.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:44.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:44.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:44.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:44.763 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:09:44.759+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:44.763367+0000) 2019-10-18T04:09:45.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:44.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:45.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:44.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:45.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:46.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:45.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:47.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:47.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:47.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:47.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:47.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:47.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:48.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:48.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:48.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:48.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:48.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:48.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:48.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:48.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:49.904 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:09:49.901+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:49.905702+0000) 2019-10-18T04:09:49.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:49.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:49.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:49.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:50.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:09:50.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:50.231236+0000) 2019-10-18T04:09:50.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:50.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:50.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:50.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:51.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:51.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:51.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:51.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:52.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:52.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:52.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:52.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:52.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:52.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:53.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:09:53.889+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:53.894147+0000) 2019-10-18T04:09:53.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:53.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:53.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:53.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:54.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:09:54.763+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:54.765470+0000) 2019-10-18T04:09:54.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:54.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:54.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:54.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:55.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:55.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:55.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:55.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:56.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:56.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:56.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:56.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:57.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:09:57.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:09:57.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:57.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:57.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:57.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:58.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:58.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:58.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:58.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:59.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:59.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:09:59.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:09:59.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:09:59.905 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:09:59.901+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:09:59.905890+0000) 2019-10-18T04:10:00.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:10:00.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:00.231341+0000) 2019-10-18T04:10:00.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:00.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:00.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:00.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:01.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:01.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:01.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:01.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:02.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:02.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:02.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:02.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:02.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:02.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:03.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:10:03.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:03.894352+0000) 2019-10-18T04:10:03.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:03.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:03.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:03.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:04.764 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:10:04.763+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:04.765568+0000) 2019-10-18T04:10:04.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:04.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:04.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:04.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:05.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:05.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:06.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:06.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:06.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:06.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:07.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:07.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:07.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:07.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:07.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:07.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:08.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:08.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:08.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:08.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:09.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:09.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:09.905 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:10:09.901+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:09.906076+0000) 2019-10-18T04:10:10.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:10:10.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:10.231439+0000) 2019-10-18T04:10:10.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:10.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:10.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:10.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:11.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:11.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:11.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:11.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:12.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:12.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:12.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:12.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:12.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:12.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:13.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:13.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:13.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:13.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:13.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:10:13.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:13.894541+0000) 2019-10-18T04:10:14.767 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:10:14.762+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:14.765667+0000) 2019-10-18T04:10:14.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:14.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:14.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:14.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:15.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:15.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:15.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:15.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:16.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:16.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:16.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:16.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:17.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:17.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:17.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:17.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:17.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:17.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:18.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:18.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:18.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:18.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:19.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:19.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:19.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:19.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:19.905 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:10:19.905+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:19.906246+0000) 2019-10-18T04:10:20.230 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:10:20.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:20.231537+0000) 2019-10-18T04:10:20.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:20.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:20.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:20.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:21.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:21.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:21.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:21.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:22.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:22.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:22.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:22.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:22.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:22.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:23.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:23.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:23.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:23.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:23.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:10:23.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:23.894678+0000) 2019-10-18T04:10:24.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:10:24.762+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:24.765769+0000) 2019-10-18T04:10:24.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:24.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:24.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:24.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:25.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:25.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:25.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:25.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:26.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:26.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:26.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:26.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:27.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:27.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:27.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:27.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:27.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:27.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:28.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:28.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:28.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:28.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:29.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:29.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:29.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:29.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:29.905 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:10:29.905+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:29.906428+0000) 2019-10-18T04:10:30.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:10:30.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:30.231636+0000) 2019-10-18T04:10:30.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:30.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:30.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:30.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:31.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:31.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:31.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:31.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:32.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:32.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:32.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:32.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:32.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:32.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:33.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:33.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:33.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:33.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:33.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:10:33.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:33.894783+0000) 2019-10-18T04:10:34.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:10:34.762+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:34.765870+0000) 2019-10-18T04:10:34.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:34.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:34.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:34.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:35.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:35.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:35.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:35.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:36.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:36.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:36.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:36.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:37.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:37.214+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:37.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:37.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:37.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:37.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:38.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:38.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:38.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:38.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:39.905 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:10:39.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:39.906607+0000) 2019-10-18T04:10:39.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:39.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:39.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:39.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:40.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:10:40.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:40.231733+0000) 2019-10-18T04:10:40.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:40.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:40.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:40.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:41.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:41.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:41.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:41.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:42.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:42.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:42.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:42.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:42.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:42.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:43.893 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:10:43.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:43.894887+0000) 2019-10-18T04:10:43.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:43.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:43.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:43.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:44.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:10:44.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:44.765968+0000) 2019-10-18T04:10:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:45.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:45.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:46.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:46.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:46.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:46.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:47.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:47.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:47.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:47.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:47.219 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:47.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:48.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:48.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:48.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:48.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:49.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:49.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:49.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:49.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:49.905 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:10:49.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:49.906787+0000) 2019-10-18T04:10:50.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:50.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:50.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:50.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:50.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:10:50.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:50.231831+0000) 2019-10-18T04:10:51.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:51.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:51.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:51.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:52.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:52.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:52.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:52.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:52.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:52.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:53.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:53.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:53.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:53.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:53.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:10:53.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:53.894989+0000) 2019-10-18T04:10:54.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:54.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:54.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:54.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:54.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:10:54.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:54.766067+0000) 2019-10-18T04:10:55.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:55.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:55.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:55.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:56.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:56.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:56.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:56.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:57.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:10:57.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:10:57.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:57.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:57.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:57.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:58.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:58.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:58.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:58.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:59.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:59.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:10:59.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:10:59.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:10:59.906 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:10:59.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:10:59.906968+0000) 2019-10-18T04:11:00.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:11:00.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:00.231935+0000) 2019-10-18T04:11:00.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:00.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:00.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:00.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:01.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:01.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:01.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:01.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:02.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:02.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:02.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:02.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:02.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:02.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:03.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:03.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:03.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:03.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:03.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:11:03.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:03.895089+0000) 2019-10-18T04:11:04.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:04.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:04.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:04.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:04.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:11:04.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:04.766169+0000) 2019-10-18T04:11:05.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:05.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:05.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:05.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:06.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:06.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:06.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:06.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:07.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:07.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:07.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:07.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:07.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:07.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:08.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:08.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:08.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:08.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:09.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:09.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:09.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:09.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:09.906 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:11:09.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:09.907150+0000) 2019-10-18T04:11:10.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:11:10.229+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:10.232031+0000) 2019-10-18T04:11:10.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:10.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:10.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:10.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:11.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:11.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:11.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:11.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:12.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:12.225+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:12.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:12.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:12.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:13.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:13.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:13.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:13.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:13.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:11:13.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:13.895184+0000) 2019-10-18T04:11:14.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:14.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:14.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:14.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:14.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:11:14.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:14.766267+0000) 2019-10-18T04:11:15.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:15.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:15.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:15.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:16.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:16.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:16.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:16.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:17.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:17.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:17.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:17.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:17.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:17.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:18.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:18.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:18.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:18.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:19.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:19.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:19.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:19.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:19.906 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:11:19.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:19.907331+0000) 2019-10-18T04:11:20.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:11:20.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:20.232126+0000) 2019-10-18T04:11:20.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:20.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:20.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:20.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:21.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:21.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:21.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:21.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:22.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:22.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:22.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:22.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:22.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:22.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:23.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:23.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:23.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:23.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:23.900 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:11:23.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:23.895297+0000) 2019-10-18T04:11:24.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:24.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:24.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:24.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:24.765 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:11:24.764+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:24.766367+0000) 2019-10-18T04:11:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:25.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:25.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:25.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:26.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:26.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:26.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:26.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:27.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:27.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:27.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:27.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:27.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:27.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:28.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:28.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:28.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:28.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:29.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:29.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:29.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:29.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:29.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:11:29.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:29.907514+0000) 2019-10-18T04:11:30.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:11:30.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:30.232225+0000) 2019-10-18T04:11:30.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:30.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:30.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:30.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:31.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:31.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:31.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:31.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:32.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:32.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:32.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:32.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:32.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:32.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:33.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:33.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:33.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:33.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:33.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:11:33.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:33.895410+0000) 2019-10-18T04:11:34.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:34.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:34.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:34.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:34.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:11:34.764+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:34.766476+0000) 2019-10-18T04:11:35.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:35.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:35.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:35.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:36.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:36.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:36.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:36.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:37.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:37.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:37.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:37.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:37.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:37.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:38.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:38.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:38.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:38.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:39.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:39.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:39.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:39.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:39.906 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:11:39.904+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:39.907697+0000) 2019-10-18T04:11:40.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:11:40.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:40.232334+0000) 2019-10-18T04:11:40.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:40.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:40.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:40.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:41.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:41.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:41.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:41.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:42.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:42.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:42.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:42.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:42.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:42.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:43.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:43.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:43.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:43.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:43.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:11:43.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:43.895508+0000) 2019-10-18T04:11:44.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:44.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:44.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:44.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:11:44.764+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:44.766574+0000) 2019-10-18T04:11:45.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:45.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:45.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:45.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:46.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:46.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:46.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:46.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:47.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:47.228+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:47.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:47.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:47.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:47.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:48.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:48.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:48.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:48.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:49.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:49.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:49.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:49.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:49.906 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:11:49.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:49.907868+0000) 2019-10-18T04:11:50.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:11:50.228+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:50.232447+0000) 2019-10-18T04:11:50.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:50.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:50.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:50.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:51.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:51.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:51.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:51.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:52.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:52.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:52.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:52.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:52.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:52.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:53.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:53.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:53.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:53.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:53.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:11:53.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:53.895602+0000) 2019-10-18T04:11:54.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:54.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:54.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:54.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 396 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:54.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:11:54.763+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:54.766674+0000) 2019-10-18T04:11:55.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:55.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:55.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:55.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:56.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:56.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:56.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:56.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:57.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:11:57.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:11:57.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:57.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:57.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:57.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:58.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:58.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:58.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:58.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:59.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:59.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:11:59.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:11:59.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:11:59.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:11:59.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:11:59.908076+0000) 2019-10-18T04:12:00.231 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:12:00.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:00.232548+0000) 2019-10-18T04:12:00.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:00.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:00.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:00.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:01.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:01.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:02.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:02.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:02.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:02.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:02.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:02.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:03.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:03.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:03.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:03.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:03.894 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:12:03.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:03.895713+0000) 2019-10-18T04:12:04.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:04.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:04.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:04.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:04.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:12:04.763+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:04.766780+0000) 2019-10-18T04:12:05.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:05.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:05.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:05.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:06.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:06.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:06.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:06.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:07.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:07.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:07.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:07.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:07.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:07.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:08.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:08.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:08.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:08.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:09.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:09.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:09.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:09.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:09.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:12:09.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:09.908259+0000) 2019-10-18T04:12:10.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:12:10.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:10.232649+0000) 2019-10-18T04:12:10.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:10.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:10.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:10.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:11.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:11.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:11.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:11.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:12.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:12.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:12.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:12.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:12.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:12.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:13.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:13.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:13.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:13.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:13.895 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:12:13.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:13.895811+0000) 2019-10-18T04:12:14.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:14.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:14.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:14.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:14.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:12:14.763+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:14.766883+0000) 2019-10-18T04:12:15.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:15.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:15.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:15.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:16.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:16.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:16.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:16.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:17.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:17.235+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:17.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:17.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:17.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:17.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:18.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:18.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:18.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:18.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:19.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:19.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:19.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:19.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:19.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:12:19.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:19.908468+0000) 2019-10-18T04:12:20.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:12:20.227+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:20.232755+0000) 2019-10-18T04:12:20.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:20.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:20.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:20.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:21.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:21.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:21.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:21.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:22.242 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:22.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:22.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:22.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:22.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:22.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:23.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:23.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:23.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:23.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:23.903 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:12:23.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:23.895905+0000) 2019-10-18T04:12:24.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:24.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:24.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:24.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:24.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:12:24.763+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:24.766987+0000) 2019-10-18T04:12:25.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:25.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:25.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:25.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:26.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:26.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:26.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:26.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:27.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:27.239+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:27.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:27.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:27.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:27.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:28.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:28.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:28.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:28.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:29.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:29.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:29.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:29.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:29.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:12:29.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:29.908650+0000) 2019-10-18T04:12:30.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:12:30.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:30.232857+0000) 2019-10-18T04:12:30.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:30.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:30.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:30.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:31.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:31.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:31.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:31.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:32.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:32.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:32.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:32.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:32.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:32.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:33.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:33.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:33.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:33.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:33.895 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:12:33.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:33.896001+0000) 2019-10-18T04:12:34.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:34.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:34.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:34.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:34.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:12:34.762+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:34.767086+0000) 2019-10-18T04:12:35.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:35.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:35.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:35.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:36.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:36.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:36.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:36.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:37.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:37.238+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:37.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:37.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:37.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:37.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:38.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:38.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:38.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:38.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:39.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:39.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:39.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:39.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:39.907 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:12:39.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:39.908830+0000) 2019-10-18T04:12:40.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:12:40.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:40.232958+0000) 2019-10-18T04:12:40.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:40.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:40.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:40.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:41.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:41.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:41.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:41.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:42.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:42.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:42.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:42.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:42.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:42.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:43.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:43.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:43.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:43.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:43.895 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:12:43.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:43.896103+0000) 2019-10-18T04:12:44.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:44.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:44.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:44.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:44.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:12:44.762+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:44.767187+0000) 2019-10-18T04:12:45.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:45.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:45.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:45.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:46.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:46.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:46.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:46.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:47.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:47.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:47.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:47.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:47.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:47.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:48.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:48.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:48.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:48.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:49.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:49.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:49.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:49.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:49.908 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:12:49.903+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:49.909013+0000) 2019-10-18T04:12:50.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:12:50.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:50.233062+0000) 2019-10-18T04:12:50.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:50.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:50.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:50.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:51.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:51.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:51.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:51.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:52.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:52.242+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:52.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:52.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:52.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:53.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:53.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:53.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:53.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:53.895 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:12:53.891+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:53.896200+0000) 2019-10-18T04:12:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:54.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:54.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:54.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:12:54.762+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:54.767288+0000) 2019-10-18T04:12:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:55.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:55.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:56.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:56.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:56.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:56.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:57.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:12:57.246+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:12:57.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:57.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:57.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:57.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:58.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:58.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:58.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:58.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:59.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:59.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:12:59.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:12:59.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:12:59.908 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:12:59.902+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:12:59.909204+0000) 2019-10-18T04:13:00.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:13:00.226+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:00.233170+0000) 2019-10-18T04:13:00.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:00.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:00.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:00.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:01.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:01.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:01.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:01.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:02.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:02.246+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:02.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:02.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:02.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:02.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:03.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:03.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:03.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:03.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:03.895 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:13:03.890+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:03.896296+0000) 2019-10-18T04:13:04.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:13:04.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:04.767393+0000) 2019-10-18T04:13:04.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:04.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:04.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:04.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:05.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:05.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:05.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:05.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:06.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:06.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:06.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:06.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:07.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:07.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:07.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:07.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:07.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:07.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:08.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:08.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:08.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:08.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:09.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:09.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:09.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:09.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:09.909 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:13:09.902+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:09.909385+0000) 2019-10-18T04:13:10.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:13:10.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:10.233273+0000) 2019-10-18T04:13:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:10.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:10.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:11.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:11.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:11.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:11.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:12.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:12.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:12.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:12.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:12.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:12.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:13.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:13.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:13.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:13.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:13.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:13:13.890+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:13.896393+0000) 2019-10-18T04:13:14.767 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:13:14.761+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:14.767496+0000) 2019-10-18T04:13:14.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:14.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:14.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:14.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:15.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:15.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:15.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:15.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:16.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:16.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:16.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:16.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:17.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:17.249+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:17.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:17.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:18.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:18.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:18.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:18.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:19.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:19.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:19.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:19.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:19.913 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:13:19.902+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:19.909589+0000) 2019-10-18T04:13:20.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:13:20.225+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:20.233384+0000) 2019-10-18T04:13:20.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:20.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:20.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:20.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:21.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:21.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:21.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:21.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:22.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:22.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:22.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:22.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:23.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:23.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:23.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:23.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:23.895 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:13:23.890+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:23.896502+0000) 2019-10-18T04:13:24.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:24.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:24.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:24.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:24.766 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:13:24.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:24.767597+0000) 2019-10-18T04:13:25.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:25.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:25.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:25.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:26.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:26.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:26.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:26.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:27.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:27.253+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:27.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:27.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:27.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:27.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:28.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:28.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:28.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:28.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:29.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:29.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:29.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:29.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:29.909 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:13:29.902+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:29.909771+0000) 2019-10-18T04:13:30.232 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:13:30.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:30.233486+0000) 2019-10-18T04:13:30.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:30.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:30.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:30.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:31.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:31.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:31.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:31.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:32.257 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:32.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:32.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:32.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:32.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:32.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:33.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:33.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:33.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:33.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:33.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:13:33.890+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:33.896602+0000) 2019-10-18T04:13:34.767 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:13:34.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:34.767697+0000) 2019-10-18T04:13:34.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:34.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:34.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:34.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:35.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:35.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:35.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:35.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:36.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:36.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:36.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:36.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:37.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:37.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:37.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:37.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:37.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:37.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:38.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:38.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:38.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:38.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:39.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:39.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:39.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:39.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:39.908 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:13:39.902+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:39.909955+0000) 2019-10-18T04:13:40.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:13:40.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:40.233585+0000) 2019-10-18T04:13:40.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:40.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:40.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:40.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:41.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:41.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:41.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:41.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:42.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:42.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:42.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:42.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:42.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:42.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:43.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:43.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:43.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:43.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:43.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:13:43.890+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:43.896713+0000) 2019-10-18T04:13:44.767 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:13:44.764+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:44.767800+0000) 2019-10-18T04:13:44.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:44.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:44.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:44.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:45.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:45.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:45.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:45.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:46.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:46.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:46.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:46.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:47.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:47.256+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:47.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:47.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:47.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:47.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:49.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:49.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:49.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:49.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:49.909 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:13:49.906+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:49.910159+0000) 2019-10-18T04:13:50.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:50.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:50.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:50.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:50.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:13:50.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:50.233689+0000) 2019-10-18T04:13:51.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:51.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:51.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:51.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:52.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:52.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:52.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:52.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:52.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:52.260+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:52.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:52.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:52.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:52.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:53.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:13:53.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:53.896823+0000) 2019-10-18T04:13:53.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:53.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:53.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:53.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:54.767 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:13:54.764+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:54.767899+0000) 2019-10-18T04:13:54.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:54.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:54.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:54.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:56.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:56.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:56.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:56.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:57.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:57.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:57.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:57.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:57.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:13:57.260+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:13:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:58.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:58.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:59.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:59.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:13:59.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:13:59.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:13:59.909 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:13:59.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:13:59.910406+0000) 2019-10-18T04:14:00.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:00.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:00.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:00.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:00.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:14:00.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:00.233797+0000) 2019-10-18T04:14:01.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:01.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:01.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:01.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:02.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:02.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:02.263 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:02.260+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:03.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:03.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:03.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:03.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:03.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:14:03.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:03.896936+0000) 2019-10-18T04:14:04.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:04.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:04.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:04.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:04.772 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:14:04.764+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:04.768001+0000) 2019-10-18T04:14:05.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:05.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:05.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:05.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:06.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:06.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:06.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:06.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:07.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:07.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:07.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:07.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:07.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:07.260+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:08.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:08.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:08.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:08.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:09.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:09.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:09.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:09.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:09.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:14:09.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:09.910609+0000) 2019-10-18T04:14:10.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:14:10.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:10.233894+0000) 2019-10-18T04:14:10.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:10.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:10.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:10.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:11.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:11.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:11.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:11.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:12.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:12.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:12.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:12.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:12.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:12.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:13.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:13.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:13.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:13.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:13.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:14:13.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:13.897049+0000) 2019-10-18T04:14:14.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:14.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:14.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:14.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:14.768 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:14:14.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:14.769470+0000) 2019-10-18T04:14:15.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:15.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:15.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:15.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:16.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:16.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:16.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:16.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:17.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:17.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:17.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:17.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:17.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:17.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:18.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:18.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:18.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:18.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:19.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:19.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:19.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:19.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:19.913 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:14:19.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:19.910817+0000) 2019-10-18T04:14:20.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:20.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:20.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:20.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:20.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:14:20.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:20.233997+0000) 2019-10-18T04:14:21.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:21.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:21.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:21.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:22.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:22.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:22.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:22.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:22.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:22.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:23.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:23.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:23.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:23.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:23.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:14:23.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:23.897160+0000) 2019-10-18T04:14:24.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:24.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:24.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:24.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:24.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:14:24.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:24.769568+0000) 2019-10-18T04:14:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:25.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:25.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:26.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:26.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:26.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:26.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:27.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:27.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:27.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:27.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:27.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:27.263+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:28.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:28.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:28.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:28.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:29.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:29.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:29.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:29.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:29.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:14:29.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:29.910993+0000) 2019-10-18T04:14:30.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:14:30.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:30.234096+0000) 2019-10-18T04:14:30.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:30.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:30.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:30.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:31.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:31.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:31.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:31.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:32.271 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:32.267+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:32.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:32.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:32.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:32.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:33.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:33.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:33.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:33.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:33.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:14:33.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:33.897275+0000) 2019-10-18T04:14:34.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:34.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:34.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:34.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:34.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:14:34.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:34.769670+0000) 2019-10-18T04:14:35.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:35.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:35.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:35.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:36.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:36.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:04:05.405820+0000 (oldest deadline 2019-10-18T04:04:25.405820+0000) 2019-10-18T04:14:36.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:36.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:37.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:37.267+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:37.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:37.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:38.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:38.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:39.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:39.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:39.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:14:39.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:39.911188+0000) 2019-10-18T04:14:40.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:14:40.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:40.234192+0000) 2019-10-18T04:14:40.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:40.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:41.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:41.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:42.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:42.267+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:42.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:42.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:43.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:43.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:43.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:14:43.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:43.897388+0000) 2019-10-18T04:14:44.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:44.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:44.771 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:14:44.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:44.769769+0000) 2019-10-18T04:14:45.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:45.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:46.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:46.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:47.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:47.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:47.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:47.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:48.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:48.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:49.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:49.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:49.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:14:49.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:49.911376+0000) 2019-10-18T04:14:50.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:14:50.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:50.234291+0000) 2019-10-18T04:14:50.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:50.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:51.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:51.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:52.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:52.270+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:52.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:52.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:53.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:53.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:53.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:14:53.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:53.897502+0000) 2019-10-18T04:14:54.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:54.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:54.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:14:54.766+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:54.769870+0000) 2019-10-18T04:14:55.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:55.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:56.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:56.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:57.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:14:57.270+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:14:57.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:57.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:14:57.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:57.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:58.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:58.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:14:58.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:58.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:59.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:59.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:14:59.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:14:59.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:14:59.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:14:59.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:14:59.911567+0000) 2019-10-18T04:15:00.233 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:15:00.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:00.234399+0000) 2019-10-18T04:15:00.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:00.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:00.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:00.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:01.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:01.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:01.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:01.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:02.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:02.270+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:02.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:02.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:02.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:02.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:03.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:03.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:03.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:03.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:03.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:15:03.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:03.897618+0000) 2019-10-18T04:15:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:04.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:04.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:04.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:04.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:15:04.766+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:04.769977+0000) 2019-10-18T04:15:05.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:05.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:05.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:05.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:06.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:06.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:06.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:06.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:07.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:07.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:07.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:07.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:07.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:07.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:08.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:08.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:08.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:08.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:09.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:09.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:09.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:09.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:09.910 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:09.909+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:09.911749+0000) 2019-10-18T04:15:10.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:15:10.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:10.234502+0000) 2019-10-18T04:15:10.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:10.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:10.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:10.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:11.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:11.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:11.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:11.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:12.280 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:12.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:12.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:12.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:12.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:12.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:13.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:13.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:13.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:13.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:13.896 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:15:13.893+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:13.897731+0000) 2019-10-18T04:15:14.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:14.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:14.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:14.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:14.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:15:14.766+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:14.770075+0000) 2019-10-18T04:15:15.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:15.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:15.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:15.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:16.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:16.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:16.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:16.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:17.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:17.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:17.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:17.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:17.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:17.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:18.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:18.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:18.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:18.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:19.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:19.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:19.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:19.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:19.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:19.908+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:19.911932+0000) 2019-10-18T04:15:20.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:15:20.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:20.234610+0000) 2019-10-18T04:15:20.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:20.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:20.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:20.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:21.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:21.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:21.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:21.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:22.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:22.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:22.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:22.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:22.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:22.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:23.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:23.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:23.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:23.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:23.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:15:23.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:23.897847+0000) 2019-10-18T04:15:24.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:24.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:24.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:24.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:24.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:15:24.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:24.770175+0000) 2019-10-18T04:15:25.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:25.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:25.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:25.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:26.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:26.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:26.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:26.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:27.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:27.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:27.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:27.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:27.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:27.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:28.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:28.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:28.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:28.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:29.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:29.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:29.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:29.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:29.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:29.908+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:29.912111+0000) 2019-10-18T04:15:30.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:15:30.229+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:30.234716+0000) 2019-10-18T04:15:30.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:30.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:30.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:30.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:31.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:31.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:31.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:32.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:32.277+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:32.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:32.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:32.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:32.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:33.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:33.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:33.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:33.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:33.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:15:33.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:33.897959+0000) 2019-10-18T04:15:34.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:34.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:34.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:34.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:34.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:15:34.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:34.770273+0000) 2019-10-18T04:15:35.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:35.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:35.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:35.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:36.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:36.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:36.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:36.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:37.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:37.281+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:37.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:37.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:37.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:37.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:38.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:38.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:38.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:38.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:39.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:39.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:39.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:39.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:39.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:39.908+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:39.912301+0000) 2019-10-18T04:15:40.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:15:40.229+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:40.234822+0000) 2019-10-18T04:15:40.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:40.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:40.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:40.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:41.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:41.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:41.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:41.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:42.286 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:42.281+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:42.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:42.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:43.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:43.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:43.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:43.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:43.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:15:43.892+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:43.898066+0000) 2019-10-18T04:15:44.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:44.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:44.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:44.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:44.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:15:44.769+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:44.770371+0000) 2019-10-18T04:15:45.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:45.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:45.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:45.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:46.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:46.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:46.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:46.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:47.287 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:47.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:47.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:47.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:47.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:47.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:48.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:48.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:48.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:48.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:49.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:49.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:49.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:49.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:49.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:49.908+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:49.912473+0000) 2019-10-18T04:15:50.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:15:50.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:50.234928+0000) 2019-10-18T04:15:50.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:50.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:50.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:50.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:51.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:51.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:51.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:51.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:52.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:52.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:52.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:52.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:52.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:52.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:53.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:53.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:53.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:53.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:53.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:15:53.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:53.898189+0000) 2019-10-18T04:15:54.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:54.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:54.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:54.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:54.769 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:15:54.769+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:54.770465+0000) 2019-10-18T04:15:55.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:55.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:55.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:55.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:56.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:56.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:56.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:56.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:57.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:15:57.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:15:57.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:57.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:57.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:57.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:58.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:58.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:58.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:58.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:59.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:59.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:15:59.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:15:59.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:15:59.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:59.908+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:59.912643+0000) 2019-10-18T04:15:59.912 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:15:59.908+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:15:59.913477+0000) 2019-10-18T04:16:00.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:16:00.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:00.235038+0000) 2019-10-18T04:16:00.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:00.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:00.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:00.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:01.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:01.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:01.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:01.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:02.290 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:02.288+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:02.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:02.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:02.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:02.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:03.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:03.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:03.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:03.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:03.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:16:03.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:03.898297+0000) 2019-10-18T04:16:04.412 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:04.408+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:04.413095+0000) 2019-10-18T04:16:04.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:04.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:04.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:04.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:04.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:16:04.768+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:04.770568+0000) 2019-10-18T04:16:05.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:05.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:05.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:05.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:06.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:06.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:06.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:06.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:07.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:07.288+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:07.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:07.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:07.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:07.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:08.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:08.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:08.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:08.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:09.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:09.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:09.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:09.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:10.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:16:10.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:10.235145+0000) 2019-10-18T04:16:10.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:10.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:10.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:10.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:11.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:11.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:11.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:11.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:12.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:12.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:12.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:12.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:12.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:12.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:13.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:13.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:13.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:13.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:13.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:16:13.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:13.898411+0000) 2019-10-18T04:16:14.412 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:14.408+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:14.413270+0000) 2019-10-18T04:16:14.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:16:14.768+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:14.770671+0000) 2019-10-18T04:16:14.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:14.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:14.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:14.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:15.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:15.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:15.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:15.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:16.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:16.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:16.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:16.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:17.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:17.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:17.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:17.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:17.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:17.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:18.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:18.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:18.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:18.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:19.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:19.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:19.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:19.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:20.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:16:20.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:20.235248+0000) 2019-10-18T04:16:20.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:20.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:20.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:20.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:21.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:21.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:21.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:21.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:22.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:22.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:22.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:22.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:22.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:22.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:23.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:16:23.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:23.898529+0000) 2019-10-18T04:16:23.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:23.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:23.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:23.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:24.413 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:24.408+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:24.413470+0000) 2019-10-18T04:16:24.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:16:24.768+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:24.770770+0000) 2019-10-18T04:16:24.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:24.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:24.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:24.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:25.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:25.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:25.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:25.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:26.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:26.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:26.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:26.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:27.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:27.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:27.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:27.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:27.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:27.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:28.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:28.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:28.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:28.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:29.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:29.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:29.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:29.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:30.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:16:30.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:30.235363+0000) 2019-10-18T04:16:30.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:30.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:30.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:30.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:31.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:31.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:31.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:31.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:32.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:32.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:32.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:32.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:32.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:32.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:33.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:33.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:33.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:33.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:33.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:16:33.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:33.898645+0000) 2019-10-18T04:16:34.412 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:34.407+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:34.413670+0000) 2019-10-18T04:16:34.413 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:34.411+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:34.414666+0000) 2019-10-18T04:16:34.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:34.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:34.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:34.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:34.777 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:16:34.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:34.770870+0000) 2019-10-18T04:16:35.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:35.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:35.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:35.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:36.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:36.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:36.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:36.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:37.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:37.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:37.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:37.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:37.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:37.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:38.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:38.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:38.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:38.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:38.913 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:38.907+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:38.914156+0000) 2019-10-18T04:16:39.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:39.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:39.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:39.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:40.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:16:40.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:40.235471+0000) 2019-10-18T04:16:40.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:40.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:40.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:40.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:41.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:41.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:41.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:41.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:42.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:42.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:42.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:42.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:42.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:42.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:43.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:43.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:43.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:43.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:43.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:16:43.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:43.898760+0000) 2019-10-18T04:16:44.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:44.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:44.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:44.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:44.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:16:44.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:44.770971+0000) 2019-10-18T04:16:45.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:45.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:45.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:45.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:46.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:46.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:46.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:46.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:47.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:47.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:47.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:47.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:47.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:47.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:48.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:48.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:48.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:48.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:48.913 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:48.911+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:48.914371+0000) 2019-10-18T04:16:49.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:49.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:49.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:49.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:50.234 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:16:50.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:50.235574+0000) 2019-10-18T04:16:50.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:50.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:50.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:50.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:51.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:51.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:51.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:51.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:52.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:52.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:52.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:52.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:52.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:52.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:53.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:53.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:53.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:53.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:53.897 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:16:53.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:53.898870+0000) 2019-10-18T04:16:54.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:54.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:54.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:54.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:54.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:16:54.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:54.771069+0000) 2019-10-18T04:16:55.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:55.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:55.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:55.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:56.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:56.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:56.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:56.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:57.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:16:57.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:16:57.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:57.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:57.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:57.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:58.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:58.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:58.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:58.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:16:58.916 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:16:58.911+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:16:58.914560+0000) 2019-10-18T04:16:59.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:59.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:16:59.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:16:59.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:00.235 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:17:00.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:00.235672+0000) 2019-10-18T04:17:00.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:00.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:00.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:00.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:01.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:01.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:01.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:01.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:02.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:02.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:02.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:02.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:02.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:02.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:03.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:03.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:03.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:03.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:03.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:17:03.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:03.898984+0000) 2019-10-18T04:17:04.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:04.767+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:04.771171+0000) 2019-10-18T04:17:04.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:04.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:04.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:04.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:05.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:05.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:05.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:05.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:06.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:06.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:06.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:06.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:07.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:07.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:07.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:07.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:07.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:07.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:08.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:08.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:08.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:08.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:08.913 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:17:08.911+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:08.914742+0000) 2019-10-18T04:17:09.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:09.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:09.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:09.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:10.235 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:17:10.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:10.235777+0000) 2019-10-18T04:17:10.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:10.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:10.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:10.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:11.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:11.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:11.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:11.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:12.306 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:12.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:12.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:12.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:12.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:12.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:13.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:13.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:13.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:13.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:13.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:17:13.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:13.899092+0000) 2019-10-18T04:17:14.771 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:14.766+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:14.771270+0000) 2019-10-18T04:17:14.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:14.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:14.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:14.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:15.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:15.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:15.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:15.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:16.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:16.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:16.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:16.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:17.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:17.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:17.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:17.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:17.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:17.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:18.914 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:17:18.911+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:18.914924+0000) 2019-10-18T04:17:18.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:18.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:18.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:18.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:19.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:19.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:19.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:19.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:20.240 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:17:20.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:20.235882+0000) 2019-10-18T04:17:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:20.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:20.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:20.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:21.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:21.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:21.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:21.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:22.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:22.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:22.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:22.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:22.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:22.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:23.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:17:23.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:23.899205+0000) 2019-10-18T04:17:23.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:23.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:23.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:23.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:24.770 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:24.766+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:24.771371+0000) 2019-10-18T04:17:24.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:24.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:24.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:24.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:25.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:25.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:25.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:25.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:26.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:26.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:26.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:26.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:27.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:27.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:27.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:27.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:27.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:27.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:28.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:28.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:28.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:28.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:28.914 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:17:28.911+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:28.915100+0000) 2019-10-18T04:17:29.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:29.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:29.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:29.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:30.235 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:17:30.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:30.235982+0000) 2019-10-18T04:17:30.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:30.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:30.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:30.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:31.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:31.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:31.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:31.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:32.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:32.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:32.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:32.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:32.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:32.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:33.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:33.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:33.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:33.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:33.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:17:33.895+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:33.899317+0000) 2019-10-18T04:17:34.771 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:34.766+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:34.771465+0000) 2019-10-18T04:17:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:34.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:34.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:35.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:35.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:35.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:35.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:36.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:36.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:36.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:36.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:37.310 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:37.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:37.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:37.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:37.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:37.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:38.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:38.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:38.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:38.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:38.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:17:38.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:38.915281+0000) 2019-10-18T04:17:39.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:39.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:39.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:39.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:40.235 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:17:40.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:40.236083+0000) 2019-10-18T04:17:40.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:40.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:40.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:40.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:41.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:41.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:41.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:41.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:42.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:42.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:42.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:42.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:42.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:42.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:43.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:43.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:43.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:43.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:43.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:17:43.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:43.899432+0000) 2019-10-18T04:17:44.771 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:44.765+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:44.771560+0000) 2019-10-18T04:17:44.772 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:44.765+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:44.772903+0000) 2019-10-18T04:17:44.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:44.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:44.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:44.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:45.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:45.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:45.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:45.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:46.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:46.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:46.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:46.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:47.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:47.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:47.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:47.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:47.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:47.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:48.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:48.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:48.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:48.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:48.915 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:17:48.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:48.915462+0000) 2019-10-18T04:17:49.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:49.265+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:49.271743+0000) 2019-10-18T04:17:49.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:49.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:49.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:49.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:50.235 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:17:50.229+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:50.236185+0000) 2019-10-18T04:17:50.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:50.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:50.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:50.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:51.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:51.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:51.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:51.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:52.314 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:52.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:52.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:52.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:52.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:52.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:53.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:53.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:53.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:53.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:53.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:17:53.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:53.899535+0000) 2019-10-18T04:17:54.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:54.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:54.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:54.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:55.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:55.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:55.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:55.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:56.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:56.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:57.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:17:57.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:17:57.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:57.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:57.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:57.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:58.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:58.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:58.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:58.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:17:58.914 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:17:58.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:58.915644+0000) 2019-10-18T04:17:59.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:17:59.265+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:17:59.271847+0000) 2019-10-18T04:17:59.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:59.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:17:59.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:17:59.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:00.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:18:00.229+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:00.236292+0000) 2019-10-18T04:18:00.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:00.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:00.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:00.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:01.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:01.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:01.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:01.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:02.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:02.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:02.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:02.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:02.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:02.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:03.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:03.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:03.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:03.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:03.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:18:03.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:03.899649+0000) 2019-10-18T04:18:04.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:04.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:04.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:04.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:05.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:05.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:05.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:05.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:06.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:06.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:06.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:06.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:07.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:07.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:07.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:07.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:07.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:07.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:08.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:08.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:08.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:08.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:08.915 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:18:08.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:08.915823+0000) 2019-10-18T04:18:09.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:18:09.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:09.271943+0000) 2019-10-18T04:18:09.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:09.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:09.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:09.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:10.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:18:10.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:10.236390+0000) 2019-10-18T04:18:10.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:10.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:10.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:10.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:11.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:11.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:11.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:11.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:12.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:12.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:12.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:12.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:12.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:12.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:13.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:13.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:13.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:13.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:13.898 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:18:13.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:13.899760+0000) 2019-10-18T04:18:14.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:14.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:14.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:14.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:15.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:15.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:15.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:15.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:16.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:16.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:16.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:16.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:17.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:17.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:17.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:17.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:17.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:17.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:18.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:18.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:18.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:18.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:18.915 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:18:18.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:18.916002+0000) 2019-10-18T04:18:19.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:18:19.268+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:19.272010+0000) 2019-10-18T04:18:19.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:19.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:19.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:19.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:20.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:18:20.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:20.236491+0000) 2019-10-18T04:18:20.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:20.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:20.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:20.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:21.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:21.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:21.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:21.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:22.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:22.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:22.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:22.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:22.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:22.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:23.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:23.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:23.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:23.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:23.900 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:18:23.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:23.899864+0000) 2019-10-18T04:18:24.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:24.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:24.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:24.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:25.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:25.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:25.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:25.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:26.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:26.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:26.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:26.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:27.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:27.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:27.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:27.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:27.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:27.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:28.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:28.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:28.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:28.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:28.915 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:18:28.910+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:28.916214+0000) 2019-10-18T04:18:29.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:18:29.268+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:29.272109+0000) 2019-10-18T04:18:29.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:29.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:29.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:29.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:30.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:18:30.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:30.236587+0000) 2019-10-18T04:18:30.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:30.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:30.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:30.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:31.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:31.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:31.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:31.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:32.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:32.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:32.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:32.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:32.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:32.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:33.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:33.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:33.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:33.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:33.900 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:18:33.894+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:33.899978+0000) 2019-10-18T04:18:34.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:34.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:34.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:34.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:35.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:35.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:35.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:35.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:36.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:36.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:36.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:36.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:37.323 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:37.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:37.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:37.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:37.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:37.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:38.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:38.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:38.915 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:18:38.914+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:38.916395+0000) 2019-10-18T04:18:39.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:18:39.268+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:39.272207+0000) 2019-10-18T04:18:39.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:39.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:39.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:39.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:40.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:18:40.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:40.236691+0000) 2019-10-18T04:18:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:40.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:40.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:41.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:41.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:41.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:41.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:42.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:42.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:42.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:42.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:42.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:42.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:43.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:43.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:43.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:43.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:43.899 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:18:43.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:43.900089+0000) 2019-10-18T04:18:44.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:44.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:44.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:44.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:45.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:45.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:45.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:45.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:46.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:46.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:46.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:46.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:47.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:47.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:47.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:47.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:47.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:47.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:48.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:48.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:48.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:48.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:48.915 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:18:48.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:48.916579+0000) 2019-10-18T04:18:49.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:18:49.268+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:49.272305+0000) 2019-10-18T04:18:49.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:49.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:49.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:49.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:50.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:18:50.232+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:50.236791+0000) 2019-10-18T04:18:50.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:50.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:50.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:50.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:51.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:51.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:51.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:51.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:52.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:52.324+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:52.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:52.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:52.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:52.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:53.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:53.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:53.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:53.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:53.899 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:18:53.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:53.900200+0000) 2019-10-18T04:18:54.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:54.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:54.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:54.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:55.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:55.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:55.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:55.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:56.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:56.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:56.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:56.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:57.327 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:18:57.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:18:57.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:57.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:57.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:57.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:58.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:58.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:58.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:58.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:18:58.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:18:58.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:58.916760+0000) 2019-10-18T04:18:59.271 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:18:59.267+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:18:59.272411+0000) 2019-10-18T04:18:59.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:59.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:18:59.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:18:59.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:00.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:19:00.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:00.236895+0000) 2019-10-18T04:19:00.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:00.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:00.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:00.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:01.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:01.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:01.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:01.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:02.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:02.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:02.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:02.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:02.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:02.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:03.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:03.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:03.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:03.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:03.899 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:19:03.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:03.900368+0000) 2019-10-18T04:19:04.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:04.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:04.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:04.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:05.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:05.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:05.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:05.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:06.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:06.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:06.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:06.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:07.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:07.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:07.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:07.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:08.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:08.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:08.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:08.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:08.916 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:19:08.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:08.916948+0000) 2019-10-18T04:19:09.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:19:09.267+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:09.272509+0000) 2019-10-18T04:19:09.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:09.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:09.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:09.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:10.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:19:10.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:10.236994+0000) 2019-10-18T04:19:10.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:10.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:10.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:10.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:11.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:11.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:11.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:11.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:12.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:12.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:12.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:12.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:12.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:12.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:13.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:13.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:13.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:13.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:13.899 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:19:13.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:13.900554+0000) 2019-10-18T04:19:14.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:14.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:14.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:14.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:15.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:15.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:15.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:15.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:16.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:16.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:16.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:16.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:17.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:17.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:17.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:17.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:17.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:17.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:18.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:18.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:18.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:18.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:18.916 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:19:18.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:18.917143+0000) 2019-10-18T04:19:19.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:19.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:19.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:19.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:19.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:19:19.267+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:19.272613+0000) 2019-10-18T04:19:20.242 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:19:20.231+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:20.237094+0000) 2019-10-18T04:19:20.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:20.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:20.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:20.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:21.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:21.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:21.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:21.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:22.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:22.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:22.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:22.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:22.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:22.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:23.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:23.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:23.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:23.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:23.899 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:19:23.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:23.900751+0000) 2019-10-18T04:19:24.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:24.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:24.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:24.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:25.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:25.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:26.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:26.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:26.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:26.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:27.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:27.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:27.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:27.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:27.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:27.331+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:28.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:28.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:28.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:28.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:28.916 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:19:28.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:28.917324+0000) 2019-10-18T04:19:29.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:29.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:29.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:29.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:29.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:19:29.267+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:29.272720+0000) 2019-10-18T04:19:30.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:30.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:30.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:30.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:30.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:19:30.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:30.237196+0000) 2019-10-18T04:19:31.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:31.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:31.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:31.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:32.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:32.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:32.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:32.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:32.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:32.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:33.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:33.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:33.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:33.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:33.900 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:19:33.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:33.900932+0000) 2019-10-18T04:19:34.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:34.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:34.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:34.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:35.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:35.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:35.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:35.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:36.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:36.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:37.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:37.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:37.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:37.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:37.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:37.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:38.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:38.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:38.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:38.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:38.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:19:38.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:38.917514+0000) 2019-10-18T04:19:39.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:39.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:39.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:39.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:39.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:19:39.266+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:39.272816+0000) 2019-10-18T04:19:40.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:40.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:40.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:40.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:40.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:19:40.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:40.237297+0000) 2019-10-18T04:19:41.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:41.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:41.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:41.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:42.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:42.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:42.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:42.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:42.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:42.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:43.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:43.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:43.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:43.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:43.900 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:19:43.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:43.901120+0000) 2019-10-18T04:19:44.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:44.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:44.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:44.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:45.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:45.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:45.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:45.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:46.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:46.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:46.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:46.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:47.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:47.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:47.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:47.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:47.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:47.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:48.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:48.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:48.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:48.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:48.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:19:48.913+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:48.917698+0000) 2019-10-18T04:19:49.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:49.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:49.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:49.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:49.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:19:49.266+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:49.273260+0000) 2019-10-18T04:19:50.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:50.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:50.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:50.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:50.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:19:50.230+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:50.237400+0000) 2019-10-18T04:19:51.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:51.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:51.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:51.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:52.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:52.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:52.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:52.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:52.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:52.334+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:53.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:53.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:53.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:53.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:53.900 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:19:53.897+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:53.901393+0000) 2019-10-18T04:19:54.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:54.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:54.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:54.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:55.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:55.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:55.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:55.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:56.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:56.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:56.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:56.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:57.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:57.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:57.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:57.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:57.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:19:57.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:19:58.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:58.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:58.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:58.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:58.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:19:58.912+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:58.917881+0000) 2019-10-18T04:19:59.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:59.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:19:59.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:19:59.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:19:59.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:19:59.266+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:19:59.273360+0000) 2019-10-18T04:20:00.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:00.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:00.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:00.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:00.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:20:00.234+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:00.237506+0000) 2019-10-18T04:20:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:01.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:01.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:01.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:02.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:02.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:02.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:02.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:02.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:02.338+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:03.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:03.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:03.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:03.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:03.901 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:20:03.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:03.901619+0000) 2019-10-18T04:20:04.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:04.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:04.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:04.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:05.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:05.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:05.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:05.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:06.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:06.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:06.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:06.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:07.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:07.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:07.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:07.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:07.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:07.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:08.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:08.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:08.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:08.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:08.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:20:08.912+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:08.918065+0000) 2019-10-18T04:20:09.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:09.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:09.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:09.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:09.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:20:09.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:09.273459+0000) 2019-10-18T04:20:10.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:10.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:10.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:10.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:10.236 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:20:10.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:10.237610+0000) 2019-10-18T04:20:11.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:11.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:11.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:11.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:12.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:12.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:12.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:12.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:12.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:12.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:13.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:13.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:13.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:13.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:13.902 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:20:13.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:13.901845+0000) 2019-10-18T04:20:14.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:14.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:14.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:14.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:15.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:15.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:15.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:15.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:16.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:16.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:16.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:16.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:17.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:17.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:17.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:17.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:17.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:17.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:18.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:18.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:18.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:18.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:18.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:20:18.916+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:18.918243+0000) 2019-10-18T04:20:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:19.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:19.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:19.272 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:20:19.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:19.273565+0000) 2019-10-18T04:20:20.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:20.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:20.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:20.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:20.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:20:20.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:20.237711+0000) 2019-10-18T04:20:21.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:21.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:21.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:21.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:22.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:22.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:22.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:22.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:22.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:22.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:23.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:23.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:23.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:23.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:23.901 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:20:23.896+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:23.902079+0000) 2019-10-18T04:20:24.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:24.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:24.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:24.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:25.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:25.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:25.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:25.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:26.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:26.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:26.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:26.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:27.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:27.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:27.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:27.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:27.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:27.349+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:28.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:28.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:28.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:28.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:28.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:20:28.916+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:28.918380+0000) 2019-10-18T04:20:29.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:29.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:29.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:29.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:29.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:20:29.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:29.273664+0000) 2019-10-18T04:20:30.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:30.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:30.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:30.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:30.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:20:30.237+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:30.237812+0000) 2019-10-18T04:20:31.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:31.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:31.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:31.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:32.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:32.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:32.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:32.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:32.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:32.349+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:33.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:33.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:33.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:33.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:33.901 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:20:33.900+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:33.902302+0000) 2019-10-18T04:20:34.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:34.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:34.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:34.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:35.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:35.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:35.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:35.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:36.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:36.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:36.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:36.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:37.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:37.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:37.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:37.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:37.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:37.349+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:38.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:38.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:38.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:38.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:38.917 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:20:38.916+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:38.918545+0000) 2019-10-18T04:20:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:39.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:39.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:39.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:20:39.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:39.273766+0000) 2019-10-18T04:20:40.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:40.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:40.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:40.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:40.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:20:40.236+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:40.237914+0000) 2019-10-18T04:20:41.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:41.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:41.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:41.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:42.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:42.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:42.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:42.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:42.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:42.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:43.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:43.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:43.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:43.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:43.901 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:20:43.900+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:43.902546+0000) 2019-10-18T04:20:44.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:44.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:44.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:44.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:45.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:45.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:45.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:45.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:46.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:46.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:46.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:46.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:47.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:47.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:47.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:47.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:47.352 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:47.348+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:48.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:48.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:48.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:48.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:48.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:20:48.916+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:48.918734+0000) 2019-10-18T04:20:49.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:49.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:49.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:49.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:49.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:20:49.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:49.273871+0000) 2019-10-18T04:20:50.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:50.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:50.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:50.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:50.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:20:50.236+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:50.238013+0000) 2019-10-18T04:20:51.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:51.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:51.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:51.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:52.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:52.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:52.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:52.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:52.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:52.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:53.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:53.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:53.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:53.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:53.901 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:20:53.900+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:53.902760+0000) 2019-10-18T04:20:54.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:54.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:54.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:54.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:55.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:55.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:56.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:56.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:56.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:56.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:57.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:57.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:57.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:57.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:57.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:20:57.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:20:58.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:58.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:58.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:58.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:58.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:20:58.916+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:58.918923+0000) 2019-10-18T04:20:59.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:59.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:20:59.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:20:59.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:20:59.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:20:59.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:20:59.273969+0000) 2019-10-18T04:21:00.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:00.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:00.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:00.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:00.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:21:00.236+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:00.238112+0000) 2019-10-18T04:21:01.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:01.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:01.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:01.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:02.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:02.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:02.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:02.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:02.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:02.352+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:03.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:03.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:03.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:03.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:03.902 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:21:03.900+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:03.903018+0000) 2019-10-18T04:21:04.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:04.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:04.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:04.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:05.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:05.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:05.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:05.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:06.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:06.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:06.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:06.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:07.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:07.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:07.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:07.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:07.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:07.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:08.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:08.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:08.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:08.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:08.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:21:08.916+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:08.919119+0000) 2019-10-18T04:21:09.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:09.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:09.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:09.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:09.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:21:09.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:09.274068+0000) 2019-10-18T04:21:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:10.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:10.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:10.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:21:10.236+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:10.238223+0000) 2019-10-18T04:21:11.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:11.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:11.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:11.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:12.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:12.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:12.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:12.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:12.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:12.356+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:13.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:13.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:13.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:13.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:13.902 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:21:13.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:13.903260+0000) 2019-10-18T04:21:14.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:14.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:14.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:14.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:15.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:15.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:15.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:15.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:16.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:16.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:16.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:16.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:17.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:17.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:17.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:17.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:17.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:17.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:18.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:18.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:18.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:18.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:18.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:21:18.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:18.919302+0000) 2019-10-18T04:21:19.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:19.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:19.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:19.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:19.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:21:19.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:19.274172+0000) 2019-10-18T04:21:20.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:20.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:20.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:20.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:20.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:21:20.235+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:20.238325+0000) 2019-10-18T04:21:21.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:21.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:21.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:21.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:22.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:22.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:22.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:22.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:22.360 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:22.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:23.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:23.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:23.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:23.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:23.904 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:21:23.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:23.903479+0000) 2019-10-18T04:21:24.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:24.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:24.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:24.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:25.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:25.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:25.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:25.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:26.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:26.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:26.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:26.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:27.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:27.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:27.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:27.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:27.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:27.355+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:28.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:28.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:28.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:28.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:28.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:21:28.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:28.919478+0000) 2019-10-18T04:21:29.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:29.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:29.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:29.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:29.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:21:29.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:29.274274+0000) 2019-10-18T04:21:30.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:30.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:30.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:30.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:30.237 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:21:30.235+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:30.238432+0000) 2019-10-18T04:21:31.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:31.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:31.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:31.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:32.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:32.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:32.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:32.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:32.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:32.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:33.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:33.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:33.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:33.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:33.902 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:21:33.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:33.903719+0000) 2019-10-18T04:21:34.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:34.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:34.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:34.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:35.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:35.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:35.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:35.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:36.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:36.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:36.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:36.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:37.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:37.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:37.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:37.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:37.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:37.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:38.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:38.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:38.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:38.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:38.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:21:38.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:38.919665+0000) 2019-10-18T04:21:39.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:39.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:39.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:39.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:39.273 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:21:39.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:39.274372+0000) 2019-10-18T04:21:40.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:40.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:40.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:40.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:40.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:21:40.235+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:40.238533+0000) 2019-10-18T04:21:41.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:41.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:41.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:41.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:41.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:41.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:41.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:41.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:42.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:42.359+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:42.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:42.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:42.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:42.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:43.903 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:21:43.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:43.903953+0000) 2019-10-18T04:21:43.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:43.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:43.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:43.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:44.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:44.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:44.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:44.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:45.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:45.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:45.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:45.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:46.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:46.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:46.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:46.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:47.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:47.363+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:47.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:47.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:47.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:47.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:48.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:48.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:48.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:48.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:48.918 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:21:48.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:48.919855+0000) 2019-10-18T04:21:49.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:21:49.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:49.274472+0000) 2019-10-18T04:21:49.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:49.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:49.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:49.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:50.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:21:50.235+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:50.238635+0000) 2019-10-18T04:21:51.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:50.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:51.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:50.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:51.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:51.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:51.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:51.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:52.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:52.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:52.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:52.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:52.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:52.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:53.903 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:21:53.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:53.904210+0000) 2019-10-18T04:21:53.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:53.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:53.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:53.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:54.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:54.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:54.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:54.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:55.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:55.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:55.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:55.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:56.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:56.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:56.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:56.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:57.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:21:57.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:21:57.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:57.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:57.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:57.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:58.919 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:21:58.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:58.920065+0000) 2019-10-18T04:21:59.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:59.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:21:59.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:21:59.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:21:59.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:21:59.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:21:59.274572+0000) 2019-10-18T04:22:00.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:00.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:00.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:00.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:00.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:22:00.234+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:00.238743+0000) 2019-10-18T04:22:01.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:01.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:01.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:01.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:02.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:02.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:02.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:02.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:02.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:02.362+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:03.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:03.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:03.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:03.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:03.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:22:03.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:03.904425+0000) 2019-10-18T04:22:04.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:04.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:04.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:04.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:05.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:05.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:05.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:05.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:06.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:06.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:06.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:06.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:07.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:07.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:07.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:07.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:07.370 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:07.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:08.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:08.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:08.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:08.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:08.919 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:08.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:08.920236+0000) 2019-10-18T04:22:09.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:09.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:09.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:09.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:09.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:22:09.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:09.274669+0000) 2019-10-18T04:22:10.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:10.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:10.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:10.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:10.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:22:10.234+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:10.238847+0000) 2019-10-18T04:22:11.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:11.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:11.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:11.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:12.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:12.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:12.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:12.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:12.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:12.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:13.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:13.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:13.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:13.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:13.903 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:22:13.899+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:13.904697+0000) 2019-10-18T04:22:14.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:14.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:14.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:14.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:15.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:15.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:15.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:15.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:16.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:16.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:16.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:16.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:17.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:17.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:17.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:17.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:17.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:17.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:18.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:18.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:18.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:18.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:18.919 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:18.915+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:18.920418+0000) 2019-10-18T04:22:18.920 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:18.915+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:18.921461+0000) 2019-10-18T04:22:19.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:19.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:19.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:19.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:19.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:22:19.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:19.274772+0000) 2019-10-18T04:22:20.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:20.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:20.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:20.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:20.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:22:20.234+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:20.238954+0000) 2019-10-18T04:22:21.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:21.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:21.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:21.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:22.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:22.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:22.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:22.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:22.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:22.366+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:23.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:23.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:23.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:23.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:23.420 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:23.414+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:23.420809+0000) 2019-10-18T04:22:23.904 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:22:23.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:23.904925+0000) 2019-10-18T04:22:24.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:24.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:24.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:24.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:25.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:25.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:25.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:25.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:26.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:26.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:26.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:26.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:27.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:27.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:27.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:27.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:27.376 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:27.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:28.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:28.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:28.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:28.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:28.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:28.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:28.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:28.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:29.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:22:29.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:29.274873+0000) 2019-10-18T04:22:29.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:29.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:29.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:29.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:30.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:22:30.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:30.239056+0000) 2019-10-18T04:22:31.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:31.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:31.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:31.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:31.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:31.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:31.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:31.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:32.376 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:32.369+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:33.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:33.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:33.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:33.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:33.420 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:33.414+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:33.420990+0000) 2019-10-18T04:22:33.904 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:22:33.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:33.905158+0000) 2019-10-18T04:22:34.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:34.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:34.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:34.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:35.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:35.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:35.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:35.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:36.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:36.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:36.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:36.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:37.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:37.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:37.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:37.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:37.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:37.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:38.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:38.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:38.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:38.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:39.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:39.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:39.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:39.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:39.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:22:39.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:39.274972+0000) 2019-10-18T04:22:40.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:40.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:40.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:40.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:40.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:22:40.233+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:40.239163+0000) 2019-10-18T04:22:41.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:41.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:41.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:41.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:42.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:42.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:42.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:42.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:42.378 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:42.373+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:43.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:43.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:43.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:43.420 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:43.414+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:43.421168+0000) 2019-10-18T04:22:43.904 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:22:43.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:43.905398+0000) 2019-10-18T04:22:44.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:44.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:44.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:44.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:45.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:45.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:46.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:46.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:46.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:46.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:47.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:47.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:47.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:47.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:47.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:47.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:48.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:48.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:48.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:48.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:49.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:49.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:49.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:49.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:49.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:22:49.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:49.275077+0000) 2019-10-18T04:22:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:50.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:50.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:50.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:50.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:22:50.237+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:50.239261+0000) 2019-10-18T04:22:51.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:51.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:51.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:51.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:52.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:52.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:52.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:52.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:52.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:52.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:53.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:53.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:53.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:53.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:53.421 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:22:53.414+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:53.421359+0000) 2019-10-18T04:22:53.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:22:53.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:53.905651+0000) 2019-10-18T04:22:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:54.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:54.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:55.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:55.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:55.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:55.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:56.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:56.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:56.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:56.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:57.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:57.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:57.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:57.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:57.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:22:57.381+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:22:58.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:58.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:58.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:58.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:59.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:59.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:22:59.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:22:59.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:22:59.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:22:59.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:22:59.275182+0000) 2019-10-18T04:23:00.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:00.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:00.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:00.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:00.238 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:00.237+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:00.239362+0000) 2019-10-18T04:23:01.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:01.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:01.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:01.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:02.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:02.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:02.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:02.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:02.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:02.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:03.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:03.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:03.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:03.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:03.420 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:23:03.414+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:03.421547+0000) 2019-10-18T04:23:03.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:23:03.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:03.905871+0000) 2019-10-18T04:23:04.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:04.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:04.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:04.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:05.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:05.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:05.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:05.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:06.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:06.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:06.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:06.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:07.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:07.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:07.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:07.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:07.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:07.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:08.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:08.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:08.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:08.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:09.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:09.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:09.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:09.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:09.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:23:09.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:09.275281+0000) 2019-10-18T04:23:10.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:10.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:10.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:10.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:10.242 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:10.236+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:10.239462+0000) 2019-10-18T04:23:11.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:11.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:11.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:11.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:12.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:12.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:12.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:12.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:12.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:12.380+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:13.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:13.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:13.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:13.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:13.421 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:23:13.414+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:13.421740+0000) 2019-10-18T04:23:13.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:23:13.898+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:13.906091+0000) 2019-10-18T04:23:14.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:14.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:14.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:14.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:15.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:15.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:15.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:15.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:16.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:16.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:16.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:16.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:16.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:16.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:16.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:16.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:17.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:17.384+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:18.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:18.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:18.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:18.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:19.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:19.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:19.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:19.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:19.274 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:23:19.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:19.275380+0000) 2019-10-18T04:23:20.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:20.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:20.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:20.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:20.239 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:20.236+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:20.239555+0000) 2019-10-18T04:23:20.239 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:20.236+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:20.240503+0000) 2019-10-18T04:23:21.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:21.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:21.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:21.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:21.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:21.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:21.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:21.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:22.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:22.384+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:22.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:22.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:22.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:22.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:23.421 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:23:23.418+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:23.421922+0000) 2019-10-18T04:23:23.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:23:23.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:23.906354+0000) 2019-10-18T04:23:24.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:24.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:24.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:24.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:24.739 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:24.736+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:24.739754+0000) 2019-10-18T04:23:25.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:25.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:25.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:25.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:26.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:26.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:26.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:26.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:27.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:27.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:27.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:27.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:27.388 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:27.384+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:28.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:28.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:28.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:28.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:29.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:29.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:29.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:29.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:29.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:23:29.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:29.275484+0000) 2019-10-18T04:23:30.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:30.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:30.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:30.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:31.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:31.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:31.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:31.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:32.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:32.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:32.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:32.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:32.389 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:32.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:33.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:33.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:33.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:33.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:33.421 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:23:33.417+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:33.422134+0000) 2019-10-18T04:23:33.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:23:33.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:33.906565+0000) 2019-10-18T04:23:34.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:34.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:34.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:34.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:34.739 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:34.736+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:34.739855+0000) 2019-10-18T04:23:35.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:35.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:35.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:35.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:36.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:36.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:36.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:36.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:37.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:37.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:37.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:37.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:37.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:37.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:38.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:38.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:38.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:38.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:39.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:39.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:39.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:39.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:39.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:23:39.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:39.275583+0000) 2019-10-18T04:23:40.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:40.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:40.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:40.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:41.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:41.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:41.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:41.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:42.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:42.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:42.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:42.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:42.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:42.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:43.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:43.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:43.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:43.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:43.421 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:23:43.421+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:43.422378+0000) 2019-10-18T04:23:43.905 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:23:43.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:43.906790+0000) 2019-10-18T04:23:44.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:44.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:44.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:44.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:44.739 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:44.735+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:44.739953+0000) 2019-10-18T04:23:45.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:45.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:45.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:45.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:46.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:46.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:46.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:46.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:47.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:47.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:47.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:47.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:47.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:47.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:48.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:48.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:48.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:48.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:48.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:48.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:49.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:23:49.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:49.275689+0000) 2019-10-18T04:23:49.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:49.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:49.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:49.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:50.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:50.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:50.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:50.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:51.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:51.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:51.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:51.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:52.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:52.387+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:52.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:52.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:52.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:52.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:53.421 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:23:53.421+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:53.422557+0000) 2019-10-18T04:23:53.906 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:23:53.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:53.907039+0000) 2019-10-18T04:23:53.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:53.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:53.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:53.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:54.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:23:54.735+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:54.740051+0000) 2019-10-18T04:23:54.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:54.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:54.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:54.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:55.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:55.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:55.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:55.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:56.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:56.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:56.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:56.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:57.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:23:57.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:23:57.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:57.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:57.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:57.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:59.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:58.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:23:59.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:23:59.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:23:59.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:23:59.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:23:59.275794+0000) 2019-10-18T04:24:00.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:00.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:00.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:00.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:00.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:00.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:00.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:00.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:02.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:02.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:02.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:02.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:02.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:02.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:02.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:02.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:02.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:02.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:03.422 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:24:03.421+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:03.422764+0000) 2019-10-18T04:24:03.906 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:24:03.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:03.907283+0000) 2019-10-18T04:24:03.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:03.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:03.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:03.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:04.739 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:24:04.735+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:04.740148+0000) 2019-10-18T04:24:04.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:04.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:04.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:04.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:05.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:05.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:05.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:05.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:06.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:06.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:06.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:06.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:07.397 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:07.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:07.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:07.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:07.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:07.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:08.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:08.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:08.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:08.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:09.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:24:09.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:09.275893+0000) 2019-10-18T04:24:10.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:10.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:10.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:10.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:10.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:10.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:10.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:10.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:11.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:11.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:11.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:11.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:12.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:12.394+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:12.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:12.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:12.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:12.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:13.422 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:24:13.421+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:13.422959+0000) 2019-10-18T04:24:13.906 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:24:13.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:13.907494+0000) 2019-10-18T04:24:13.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:13.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:13.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:13.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:14.739 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:24:14.734+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:14.740248+0000) 2019-10-18T04:24:14.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:14.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:14.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:14.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:15.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:15.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:15.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:15.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:16.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:16.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:16.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:16.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:17.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:17.394+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:17.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:17.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:17.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:17.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:18.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:18.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:18.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:18.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:19.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:24:19.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:19.275997+0000) 2019-10-18T04:24:19.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:19.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:19.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:19.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:20.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:20.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:20.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:20.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:22.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:22.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:22.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:22.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:22.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:22.394+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:22.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:22.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:22.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:22.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:23.422 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:24:23.421+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:23.423139+0000) 2019-10-18T04:24:23.906 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:24:23.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:23.907715+0000) 2019-10-18T04:24:24.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:24.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:24.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:24.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:24.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:24:24.734+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:24.740342+0000) 2019-10-18T04:24:25.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:25.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:25.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:25.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:25.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:25.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:25.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:26.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:26.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:26.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:26.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:27.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:27.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:27.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:27.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:27.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:27.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:28.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:28.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:28.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:28.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:29.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:24:29.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:29.276099+0000) 2019-10-18T04:24:29.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:29.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:29.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:29.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:30.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:30.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:30.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:30.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:32.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:32.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:32.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:32.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:32.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:32.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:33.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:33.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:33.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:33.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:33.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:24:33.421+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:33.423360+0000) 2019-10-18T04:24:33.907 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:24:33.905+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:33.907956+0000) 2019-10-18T04:24:34.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:34.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:34.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:34.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:34.739 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:24:34.734+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:34.740441+0000) 2019-10-18T04:24:35.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:35.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:35.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:35.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:36.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:36.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:36.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:36.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:37.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:37.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:37.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:37.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:37.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:37.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:38.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:38.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:38.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:38.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:39.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:39.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:39.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:39.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:39.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:24:39.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:39.276205+0000) 2019-10-18T04:24:40.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:40.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:40.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:40.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:41.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:41.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:41.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:41.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:42.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:42.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:42.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:42.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:42.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:42.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:43.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:43.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:43.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:43.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:43.422 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:24:43.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:43.423548+0000) 2019-10-18T04:24:43.907 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:24:43.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:43.908173+0000) 2019-10-18T04:24:44.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:44.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:44.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:44.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:44.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:24:44.734+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:44.740539+0000) 2019-10-18T04:24:45.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:45.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:45.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:45.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:46.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:46.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:46.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:46.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:47.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:47.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:47.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:47.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:48.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:48.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:48.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:48.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:49.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:49.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:49.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:49.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:49.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:24:49.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:49.276316+0000) 2019-10-18T04:24:50.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:50.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:50.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:50.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:51.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:51.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:51.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:51.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:52.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:52.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:52.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:52.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:52.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:52.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:53.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:53.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:53.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:53.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:53.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:24:53.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:53.423734+0000) 2019-10-18T04:24:53.907 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:24:53.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:53.908414+0000) 2019-10-18T04:24:54.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:54.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:54.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:54.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:54.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:24:54.733+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:54.740639+0000) 2019-10-18T04:24:55.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:55.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:55.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:55.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:56.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:56.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:56.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:56.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:57.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:57.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:57.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:57.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:57.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:24:57.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:24:58.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:58.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:58.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:58.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:59.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:59.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:24:59.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:24:59.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:24:59.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:24:59.269+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:24:59.276419+0000) 2019-10-18T04:25:00.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:00.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:00.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:00.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:01.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:01.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:01.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:01.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:02.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:02.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:02.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:02.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:02.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:02.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:03.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:03.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:03.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:03.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:03.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:25:03.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:03.423919+0000) 2019-10-18T04:25:03.907 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:25:03.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:03.908629+0000) 2019-10-18T04:25:04.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:04.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:04.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:04.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:04.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:25:04.733+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:04.740738+0000) 2019-10-18T04:25:05.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:05.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:05.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:05.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:06.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:06.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:06.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:06.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:07.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:07.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:07.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:07.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:07.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:07.405+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:08.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:08.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:08.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:08.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:09.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:09.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:09.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:09.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:09.275 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:25:09.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:09.276518+0000) 2019-10-18T04:25:10.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:10.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:14:37.240754+0000 (oldest deadline 2019-10-18T04:14:57.240754+0000) 2019-10-18T04:25:10.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:10.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:11.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:11.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:12.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:12.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:12.410 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:12.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:13.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:13.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:13.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:25:13.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:13.424111+0000) 2019-10-18T04:25:13.908 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:25:13.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:13.908866+0000) 2019-10-18T04:25:14.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:14.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:14.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:25:14.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:14.740842+0000) 2019-10-18T04:25:15.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:15.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:16.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:16.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:17.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:17.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:17.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:17.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:18.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:18.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:19.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:25:19.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:19.276617+0000) 2019-10-18T04:25:19.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:19.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:20.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:20.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:21.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:21.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:22.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:22.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:22.412 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:22.408+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:23.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:23.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:23.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:25:23.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:23.424310+0000) 2019-10-18T04:25:23.908 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:25:23.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:23.909101+0000) 2019-10-18T04:25:24.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:24.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:24.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:25:24.736+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:24.740944+0000) 2019-10-18T04:25:25.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:25.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:26.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:26.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:27.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:27.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:27.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:27.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:28.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:28.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:29.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:25:29.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:29.276721+0000) 2019-10-18T04:25:29.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:29.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:30.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:30.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:31.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:31.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:31.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:31.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:32.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:32.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:32.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:32.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:32.415 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:32.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:33.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:33.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:33.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:33.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:33.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:25:33.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:33.424523+0000) 2019-10-18T04:25:33.908 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:25:33.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:33.909317+0000) 2019-10-18T04:25:34.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:34.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:34.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:34.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:34.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:25:34.736+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:34.741045+0000) 2019-10-18T04:25:35.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:35.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:35.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:35.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:36.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:36.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:36.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:36.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:37.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:37.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:37.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:37.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:37.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:37.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:38.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:38.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:38.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:38.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:39.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:25:39.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:39.276823+0000) 2019-10-18T04:25:39.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:39.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:39.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:39.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:40.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:40.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:40.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:40.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:41.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:41.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:41.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:41.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:42.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:42.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:42.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:42.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:42.418 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:42.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:43.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:43.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:43.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:43.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:43.423 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:25:43.420+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:43.424710+0000) 2019-10-18T04:25:43.908 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:25:43.904+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:43.909556+0000) 2019-10-18T04:25:44.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:44.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:44.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:44.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:44.740 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:25:44.736+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:44.741146+0000) 2019-10-18T04:25:45.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:45.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:45.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:45.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:46.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:46.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:46.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:46.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:47.418 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:47.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:47.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:47.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:47.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:47.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:48.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:48.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:48.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:48.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:49.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:25:49.272+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:49.276928+0000) 2019-10-18T04:25:49.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:49.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:49.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:49.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:50.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:50.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:50.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:50.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:51.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:51.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:51.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:51.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:52.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:52.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:52.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:52.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:52.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:52.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:53.424 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:25:53.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:53.424898+0000) 2019-10-18T04:25:53.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:53.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:53.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:53.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:53.909 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:25:53.903+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:53.909782+0000) 2019-10-18T04:25:54.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:54.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:54.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:54.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:54.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:25:54.736+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:54.741246+0000) 2019-10-18T04:25:55.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:55.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:55.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:55.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:56.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:56.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:56.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:56.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:57.421 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:25:57.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:25:57.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:57.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:57.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:57.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:58.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:58.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:58.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:58.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:25:59.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:25:59.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:25:59.277027+0000) 2019-10-18T04:25:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:59.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:25:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:25:59.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:00.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:00.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:00.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:01.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:01.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:01.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:01.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:02.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:02.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:02.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:02.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:02.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:02.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:03.424 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:26:03.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:03.425082+0000) 2019-10-18T04:26:03.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:03.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:03.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:03.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:03.909 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:26:03.903+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:03.909995+0000) 2019-10-18T04:26:04.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:04.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:04.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:04.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:04.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:26:04.735+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:04.741353+0000) 2019-10-18T04:26:05.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:05.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:05.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:05.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:06.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:06.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:06.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:06.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:07.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:07.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:07.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:07.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:07.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:07.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:08.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:08.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:08.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:08.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:09.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:26:09.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:09.277129+0000) 2019-10-18T04:26:09.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:09.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:09.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:09.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:10.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:10.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:10.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:10.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:11.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:11.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:11.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:11.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:12.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:12.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:12.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:12.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:12.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:12.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:13.427 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:26:13.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:13.425271+0000) 2019-10-18T04:26:13.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:13.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:13.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:13.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:13.909 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:26:13.907+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:13.910238+0000) 2019-10-18T04:26:14.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:14.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:14.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:14.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:14.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:26:14.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:14.741455+0000) 2019-10-18T04:26:15.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:15.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:15.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:15.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:16.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:16.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:16.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:16.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:17.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:17.423+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:17.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:17.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:17.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:17.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:18.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:18.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:18.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:18.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:19.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:26:19.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:19.277229+0000) 2019-10-18T04:26:19.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:19.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:19.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:19.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:20.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:20.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:20.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:20.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:21.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:21.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:21.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:21.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:22.427 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:22.423+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:22.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:22.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:22.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:22.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:23.424 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:26:23.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:23.425455+0000) 2019-10-18T04:26:23.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:23.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:23.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:23.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:23.909 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:26:23.907+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:23.910481+0000) 2019-10-18T04:26:24.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:24.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:24.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:24.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:24.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:26:24.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:24.741553+0000) 2019-10-18T04:26:25.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:25.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:25.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:25.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:26.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:26.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:26.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:26.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:27.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:27.423+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:27.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:27.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:27.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:27.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:28.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:28.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:28.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:28.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:29.276 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:26:29.271+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:29.277328+0000) 2019-10-18T04:26:29.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:29.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:29.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:29.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:30.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:30.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:30.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:30.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:31.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:31.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:31.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:31.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:32.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:32.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:32.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:32.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:32.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:32.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:33.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:33.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:33.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:33.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:33.425 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:26:33.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:33.425644+0000) 2019-10-18T04:26:33.909 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:26:33.907+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:33.910683+0000) 2019-10-18T04:26:34.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:34.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:34.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:34.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:34.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:26:34.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:34.741652+0000) 2019-10-18T04:26:35.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:35.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:35.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:35.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:36.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:36.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:36.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:36.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:37.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:37.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:37.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:37.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:37.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:37.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:38.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:38.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:38.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:38.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:39.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:26:39.270+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:39.277427+0000) 2019-10-18T04:26:39.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:39.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:39.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:39.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:40.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:40.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:41.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:41.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:42.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:42.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:42.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:42.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:42.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:43.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:43.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:43.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:43.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:43.425 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:26:43.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:43.425834+0000) 2019-10-18T04:26:43.910 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:26:43.907+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:43.910877+0000) 2019-10-18T04:26:44.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:44.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:44.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:44.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:44.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:26:44.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:44.741752+0000) 2019-10-18T04:26:45.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:45.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:45.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:45.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:46.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:46.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:46.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:46.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:47.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:47.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:47.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:47.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:47.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:47.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:48.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:48.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:48.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:48.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:49.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:26:49.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:49.277537+0000) 2019-10-18T04:26:49.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:49.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:49.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:49.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:50.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:50.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:50.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:50.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:51.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:51.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:52.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:52.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:52.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:52.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:52.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:52.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:53.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:53.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:53.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:53.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:53.425 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:26:53.419+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:53.426030+0000) 2019-10-18T04:26:53.910 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:26:53.907+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:53.911126+0000) 2019-10-18T04:26:54.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:54.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:54.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:54.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 408 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:54.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:26:54.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:54.741849+0000) 2019-10-18T04:26:55.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:55.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:55.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:55.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:56.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:56.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:56.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:56.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:57.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:57.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:57.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:57.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:57.444 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:26:57.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:26:58.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:58.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:58.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:58.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:26:59.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:26:59.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:26:59.277637+0000) 2019-10-18T04:26:59.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:59.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:26:59.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:26:59.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:00.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:00.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:00.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:00.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:01.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:01.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:01.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:01.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:02.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:02.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:02.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:02.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:02.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:02.430+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:03.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:03.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:03.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:03.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:03.425 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:27:03.422+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:03.426224+0000) 2019-10-18T04:27:03.910 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:27:03.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:03.911343+0000) 2019-10-18T04:27:04.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:04.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:04.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:04.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:04.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:27:04.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:04.741945+0000) 2019-10-18T04:27:05.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:05.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:05.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:05.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:06.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:06.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:06.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:06.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:07.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:07.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:07.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:07.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:07.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:07.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:08.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:08.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:08.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:08.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:09.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:27:09.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:09.277733+0000) 2019-10-18T04:27:09.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:09.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:09.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:09.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:10.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:10.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:10.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:10.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:11.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:11.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:11.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:11.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:12.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:12.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:12.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:12.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:12.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:12.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:13.425 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:27:13.422+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:13.426404+0000) 2019-10-18T04:27:13.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:13.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:13.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:13.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:13.911 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:27:13.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:13.911540+0000) 2019-10-18T04:27:14.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:14.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:14.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:14.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:14.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:27:14.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:14.742047+0000) 2019-10-18T04:27:15.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:15.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:15.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:15.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:16.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:16.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:16.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:16.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:17.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:17.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:17.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:17.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:17.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:17.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:18.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:18.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:18.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:18.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:19.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:27:19.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:19.277834+0000) 2019-10-18T04:27:19.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:19.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:19.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:19.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:20.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:20.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:20.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:20.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:21.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:21.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:21.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:21.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:22.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:22.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:22.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:22.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:22.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:22.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:23.427 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:27:23.422+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:23.426590+0000) 2019-10-18T04:27:23.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:23.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:23.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:23.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:23.910 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:27:23.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:23.911750+0000) 2019-10-18T04:27:24.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:24.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:24.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:24.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:24.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:27:24.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:24.742142+0000) 2019-10-18T04:27:25.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:25.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:25.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:25.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:26.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:26.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:26.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:26.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:27.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:27.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:27.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:27.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:27.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:27.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:28.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:28.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:28.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:28.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:29.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:27:29.277+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:29.277930+0000) 2019-10-18T04:27:29.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:29.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:29.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:29.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:30.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:30.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:30.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:30.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:31.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:31.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:31.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:31.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:32.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:32.441+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:32.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:32.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:32.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:32.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:33.426 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:27:33.422+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:33.426776+0000) 2019-10-18T04:27:33.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:33.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:33.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:33.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:33.911 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:27:33.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:33.911967+0000) 2019-10-18T04:27:34.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:34.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:34.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:34.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:34.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:27:34.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:34.742237+0000) 2019-10-18T04:27:35.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:35.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:35.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:35.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:36.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:36.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:36.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:36.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:37.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:37.441+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:37.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:37.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:37.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:37.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:38.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:38.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:38.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:38.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:39.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:27:39.277+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:39.278027+0000) 2019-10-18T04:27:39.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:39.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:39.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:39.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:40.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:40.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:40.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:40.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:41.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:41.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:41.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:41.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:42.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:42.445+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:42.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:42.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:42.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:42.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:43.426 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:27:43.422+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:43.426963+0000) 2019-10-18T04:27:43.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:43.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:43.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:43.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:43.912 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:27:43.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:43.912182+0000) 2019-10-18T04:27:44.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:44.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:44.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:44.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:44.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:27:44.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:44.742338+0000) 2019-10-18T04:27:45.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:45.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:45.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:45.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:46.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:46.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:46.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:46.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:47.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:47.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:47.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:47.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:47.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:47.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:48.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:48.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:48.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:48.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:49.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:27:49.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:49.278125+0000) 2019-10-18T04:27:49.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:49.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:49.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:49.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:50.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:50.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:50.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:50.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:51.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:51.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:52.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:52.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:52.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:52.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:52.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:52.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:53.426 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:27:53.422+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:53.427173+0000) 2019-10-18T04:27:53.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:53.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:53.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:53.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:53.911 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:27:53.906+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:53.912389+0000) 2019-10-18T04:27:54.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:54.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:54.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:54.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:54.741 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:27:54.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:54.742437+0000) 2019-10-18T04:27:55.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:55.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:55.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:55.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:56.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:56.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:56.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:56.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:57.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:27:57.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:27:57.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:57.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:57.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:57.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:58.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:58.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:58.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:27:59.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:27:59.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:27:59.278226+0000) 2019-10-18T04:27:59.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:59.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:27:59.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:27:59.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:00.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:00.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:00.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:00.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:01.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:01.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:01.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:01.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:02.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:02.444+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:02.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:02.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:02.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:02.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:03.426 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:28:03.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:03.427375+0000) 2019-10-18T04:28:03.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:03.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:03.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:03.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:03.917 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:28:03.910+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:03.912641+0000) 2019-10-18T04:28:04.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:04.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:04.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:04.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:04.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:28:04.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:04.742534+0000) 2019-10-18T04:28:05.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:05.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:05.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:05.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:06.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:06.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:06.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:06.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:07.450 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:07.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:07.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:07.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:07.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:07.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:08.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:08.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:08.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:08.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:09.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:28:09.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:09.278323+0000) 2019-10-18T04:28:09.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:09.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:09.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:09.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:10.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:10.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:10.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:10.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:11.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:11.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:11.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:11.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:12.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:12.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:12.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:12.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:12.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:12.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:13.426 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:28:13.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:13.427566+0000) 2019-10-18T04:28:13.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:13.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:13.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:13.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:13.917 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:28:13.909+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:13.912855+0000) 2019-10-18T04:28:14.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:14.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:14.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:14.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:14.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:28:14.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:14.742631+0000) 2019-10-18T04:28:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:15.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:15.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:16.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:16.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:16.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:16.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:17.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:17.448+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:17.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:17.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:17.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:17.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:18.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:18.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:18.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:18.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:19.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:28:19.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:19.278426+0000) 2019-10-18T04:28:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:19.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:19.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:20.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:20.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:20.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:20.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:21.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:21.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:21.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:21.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:22.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:22.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:22.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:22.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:22.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:22.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:23.426 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:28:23.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:23.427747+0000) 2019-10-18T04:28:23.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:23.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:23.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:23.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:23.912 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:28:23.909+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:23.913069+0000) 2019-10-18T04:28:24.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:24.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:24.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:24.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:24.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:28:24.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:24.742728+0000) 2019-10-18T04:28:25.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:25.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:25.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:25.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:26.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:26.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:26.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:26.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:27.455 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:27.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:27.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:27.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:27.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:27.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:28.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:28.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:28.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:28.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:29.277 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:28:29.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:29.278523+0000) 2019-10-18T04:28:29.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:29.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:29.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:29.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:30.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:30.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:30.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:30.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:31.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:31.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:31.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:31.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:32.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:32.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:32.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:32.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:32.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:32.451+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:33.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:33.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:33.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:33.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:33.427 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:28:33.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:33.427935+0000) 2019-10-18T04:28:33.912 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:28:33.909+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:33.913281+0000) 2019-10-18T04:28:34.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:34.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:34.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:34.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:34.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:28:34.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:34.742825+0000) 2019-10-18T04:28:35.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:35.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:35.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:35.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:36.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:36.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:36.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:36.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:37.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:37.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:37.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:37.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:37.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:37.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:38.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:38.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:38.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:38.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:39.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:28:39.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:39.278621+0000) 2019-10-18T04:28:39.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:39.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:39.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:39.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:40.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:40.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:40.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:40.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:41.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:41.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:41.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:41.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:42.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:42.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:42.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:42.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:42.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:42.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:43.427 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:28:43.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:43.428125+0000) 2019-10-18T04:28:43.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:43.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:43.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:43.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:43.912 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:28:43.909+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:43.913526+0000) 2019-10-18T04:28:44.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:44.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:44.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:44.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:44.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:28:44.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:44.742925+0000) 2019-10-18T04:28:45.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:45.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:45.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:45.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:46.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:46.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:46.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:46.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:47.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:47.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:47.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:47.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:47.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:47.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:48.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:48.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:48.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:48.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:49.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:28:49.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:49.278725+0000) 2019-10-18T04:28:49.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:49.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:49.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:49.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:50.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:50.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:50.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:50.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:51.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:51.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:51.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:51.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:52.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:52.455+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:52.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:52.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:52.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:53.427 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:28:53.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:53.428309+0000) 2019-10-18T04:28:53.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:53.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:53.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:53.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:53.912 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:28:53.909+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:53.913737+0000) 2019-10-18T04:28:54.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:54.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:54.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:54.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:54.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:28:54.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:54.743023+0000) 2019-10-18T04:28:55.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:55.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:55.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:55.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:56.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:56.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:56.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:56.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:57.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:57.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:57.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:57.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:57.460 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:28:57.454+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:28:58.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:58.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:58.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:58.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:28:59.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:28:59.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:28:59.278822+0000) 2019-10-18T04:28:59.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:59.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:28:59.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:28:59.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:00.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:00.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:00.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:00.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:01.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:01.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:01.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:01.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:02.462 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:02.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:02.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:02.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:02.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:02.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:03.427 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:29:03.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:03.428492+0000) 2019-10-18T04:29:03.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:03.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:03.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:03.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:03.913 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:29:03.909+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:03.913952+0000) 2019-10-18T04:29:04.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:04.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:04.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:04.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:04.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:29:04.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:04.743120+0000) 2019-10-18T04:29:05.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:05.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:05.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:05.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:06.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:06.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:06.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:06.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:07.462 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:07.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:07.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:07.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:07.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:07.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:08.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:08.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:08.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:08.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:09.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:29:09.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:09.278919+0000) 2019-10-18T04:29:09.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:09.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:09.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:09.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:10.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:10.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:10.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:10.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:11.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:11.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:11.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:11.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:12.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:12.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:12.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:12.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:12.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:12.458+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:13.428 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:29:13.425+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:13.428678+0000) 2019-10-18T04:29:13.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:13.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:13.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:13.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:13.913 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:29:13.913+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:13.914209+0000) 2019-10-18T04:29:14.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:14.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:14.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:14.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:14.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:29:14.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:14.743221+0000) 2019-10-18T04:29:15.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:15.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:15.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:15.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:16.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:16.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:16.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:16.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:17.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:17.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:17.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:17.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:17.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:17.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:18.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:18.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:18.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:18.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:19.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:29:19.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:19.279021+0000) 2019-10-18T04:29:19.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:19.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:19.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:19.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:20.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:20.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:21.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:21.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:21.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:21.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:22.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:22.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:22.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:22.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:22.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:22.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:23.428 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:29:23.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:23.428879+0000) 2019-10-18T04:29:23.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:23.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:23.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:23.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:23.913 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:29:23.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:23.914429+0000) 2019-10-18T04:29:24.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:24.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:24.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:24.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:24.742 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:29:24.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:24.743317+0000) 2019-10-18T04:29:25.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:25.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:25.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:25.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:26.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:26.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:26.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:26.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:27.472 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:27.462+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:27.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:27.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:27.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:27.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:28.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:28.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:28.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:28.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:29.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:29:29.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:29.279117+0000) 2019-10-18T04:29:29.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:29.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:29.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:29.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:30.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:30.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:30.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:30.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:31.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:31.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:31.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:31.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:32.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:32.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:32.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:32.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:32.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:32.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:33.428 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:29:33.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:33.429061+0000) 2019-10-18T04:29:33.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:33.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:33.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:33.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:33.913 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:29:33.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:33.914647+0000) 2019-10-18T04:29:34.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:34.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:34.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:34.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:34.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:29:34.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:34.743415+0000) 2019-10-18T04:29:35.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:35.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:35.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:35.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:36.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:36.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:36.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:36.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:37.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:37.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:37.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:37.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:37.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:37.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:38.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:38.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:38.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:38.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:39.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:29:39.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:39.279218+0000) 2019-10-18T04:29:39.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:39.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:39.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:39.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:40.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:40.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:40.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:40.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:41.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:41.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:41.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:41.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:42.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:42.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:42.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:42.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:42.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:42.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:43.428 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:29:43.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:43.429248+0000) 2019-10-18T04:29:43.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:43.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:43.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:43.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:43.914 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:29:43.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:43.914882+0000) 2019-10-18T04:29:44.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:44.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:44.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:44.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:44.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:29:44.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:44.743517+0000) 2019-10-18T04:29:45.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:45.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:45.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:45.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:46.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:46.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:46.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:46.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:47.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:47.465+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:47.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:47.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:47.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:47.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:48.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:48.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:48.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:48.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:49.278 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:29:49.277+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:49.279324+0000) 2019-10-18T04:29:49.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:49.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:49.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:49.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:50.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:50.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:50.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:50.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:51.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:51.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:51.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:51.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:52.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:52.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:52.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:52.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:52.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:52.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:53.429 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:29:53.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:53.429430+0000) 2019-10-18T04:29:53.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:53.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:53.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:53.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:53.914 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:29:53.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:53.915138+0000) 2019-10-18T04:29:54.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:54.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:54.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:54.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:54.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:29:54.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:54.743613+0000) 2019-10-18T04:29:55.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:55.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:55.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:55.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:56.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:56.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:56.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:56.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:57.474 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:29:57.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:29:57.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:57.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:57.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:57.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:58.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:58.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:58.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:58.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:29:59.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:29:59.277+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:29:59.279422+0000) 2019-10-18T04:29:59.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:59.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:29:59.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:29:59.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:00.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:00.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:00.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:00.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:01.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:01.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:01.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:01.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:02.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:02.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:02.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:02.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:02.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:02.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:03.428 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:30:03.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:03.429611+0000) 2019-10-18T04:30:03.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:03.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:03.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:03.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:03.914 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:30:03.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:03.915364+0000) 2019-10-18T04:30:04.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:04.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:04.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:04.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:04.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:30:04.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:04.743709+0000) 2019-10-18T04:30:05.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:05.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:05.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:05.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:06.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:06.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:06.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:06.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:07.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:07.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:07.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:07.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:08.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:08.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:09.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:30:09.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:09.279521+0000) 2019-10-18T04:30:09.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:09.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:09.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:09.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:10.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:10.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:10.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:10.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:11.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:11.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:11.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:11.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:12.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:12.472+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:12.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:12.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:12.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:12.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:13.429 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:30:13.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:13.429798+0000) 2019-10-18T04:30:13.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:13.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:13.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:13.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:13.915 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:30:13.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:13.915583+0000) 2019-10-18T04:30:14.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:14.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:14.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:14.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:14.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:30:14.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:14.743808+0000) 2019-10-18T04:30:15.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:15.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:15.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:15.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:16.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:16.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:16.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:16.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:17.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:17.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:17.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:17.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:17.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:17.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:18.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:18.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:18.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:18.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:19.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:30:19.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:19.279619+0000) 2019-10-18T04:30:19.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:19.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:19.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:19.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:20.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:20.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:20.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:20.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:21.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:21.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:21.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:21.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:22.479 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:22.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:22.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:22.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:22.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:22.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:23.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:30:23.424+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:23.429986+0000) 2019-10-18T04:30:23.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:23.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:23.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:23.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:23.915 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:30:23.912+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:23.915830+0000) 2019-10-18T04:30:24.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:24.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:24.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:24.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:24.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:30:24.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:24.743909+0000) 2019-10-18T04:30:25.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:25.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:25.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:25.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:26.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:26.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:26.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:26.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:27.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:27.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:27.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:27.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:27.476+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:28.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:28.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:28.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:28.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:29.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:30:29.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:29.279718+0000) 2019-10-18T04:30:29.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:29.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:29.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:29.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:30.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:30.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:30.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:30.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:31.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:31.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:31.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:31.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:32.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:32.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:32.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:32.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:32.481 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:32.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:33.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:33.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:33.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:33.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:33.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:30:33.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:33.430174+0000) 2019-10-18T04:30:33.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:30:33.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:33.916044+0000) 2019-10-18T04:30:34.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:34.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:34.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:34.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:34.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:30:34.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:34.744007+0000) 2019-10-18T04:30:35.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:35.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:35.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:35.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:36.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:36.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:36.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:36.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:37.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:37.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:37.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:37.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:37.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:37.480+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:38.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:38.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:38.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:38.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:39.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:30:39.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:39.279815+0000) 2019-10-18T04:30:39.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:39.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:39.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:39.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:40.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:40.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:40.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:40.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:41.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:41.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:41.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:41.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:42.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:42.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:42.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:42.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:42.484 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:42.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:43.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:43.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:43.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:43.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:43.432 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:30:43.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:43.430380+0000) 2019-10-18T04:30:43.915 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:30:43.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:43.916268+0000) 2019-10-18T04:30:44.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:44.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:44.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:44.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:44.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:30:44.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:44.744123+0000) 2019-10-18T04:30:45.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:45.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:45.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:45.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:46.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:46.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:46.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:46.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:47.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:47.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:47.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:47.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:47.484 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:47.479+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:48.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:48.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:48.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:48.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:49.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:49.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:49.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:49.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:49.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:30:49.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:49.279916+0000) 2019-10-18T04:30:50.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:50.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:50.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:50.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:51.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:51.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:51.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:51.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:52.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:52.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:52.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:52.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:52.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:52.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:53.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:53.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:53.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:53.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:53.429 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:30:53.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:53.430575+0000) 2019-10-18T04:30:53.915 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:30:53.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:53.916487+0000) 2019-10-18T04:30:54.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:54.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:54.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:54.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:54.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:30:54.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:54.744222+0000) 2019-10-18T04:30:55.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:55.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:55.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:55.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:56.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:56.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:56.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:56.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:57.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:57.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:57.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:57.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:57.486 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:30:57.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:30:58.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:58.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:58.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:58.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:59.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:59.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:30:59.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:30:59.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:30:59.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:30:59.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:30:59.280014+0000) 2019-10-18T04:31:00.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:00.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:00.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:00.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:01.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:01.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:01.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:01.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:02.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:02.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:02.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:02.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:02.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:02.483+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:03.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:03.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:03.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:03.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:03.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:31:03.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:03.430766+0000) 2019-10-18T04:31:03.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:31:03.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:03.916718+0000) 2019-10-18T04:31:04.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:04.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:04.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:04.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:04.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:31:04.739+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:04.744319+0000) 2019-10-18T04:31:05.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:05.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:05.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:05.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:06.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:06.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:06.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:06.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:07.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:07.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:07.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:07.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:07.489 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:07.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:08.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:08.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:08.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:08.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:09.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:09.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:09.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:09.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:09.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:31:09.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:09.280115+0000) 2019-10-18T04:31:10.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:10.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:10.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:10.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:11.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:11.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:11.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:11.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:12.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:12.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:12.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:12.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:12.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:12.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:13.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:13.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:13.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:13.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:13.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:31:13.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:13.430956+0000) 2019-10-18T04:31:13.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:31:13.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:13.916958+0000) 2019-10-18T04:31:14.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:14.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:14.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:14.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:14.743 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:31:14.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:14.744424+0000) 2019-10-18T04:31:15.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:15.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:15.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:15.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:16.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:16.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:16.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:16.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:17.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:17.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:17.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:17.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:17.491 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:17.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:18.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:18.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:18.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:18.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:19.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:19.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:19.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:19.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:19.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:31:19.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:19.280221+0000) 2019-10-18T04:31:20.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:20.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:20.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:20.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:21.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:21.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:21.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:21.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:22.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:22.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:22.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:22.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:22.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:22.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:23.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:22.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:23.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:22.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:23.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:31:23.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:23.431167+0000) 2019-10-18T04:31:23.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:31:23.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:23.917186+0000) 2019-10-18T04:31:24.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:24.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:24.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:24.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:24.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:31:24.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:24.744524+0000) 2019-10-18T04:31:25.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:25.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:25.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:25.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:26.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:26.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:26.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:26.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:27.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:27.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:27.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:27.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:27.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:27.486+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:28.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:28.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:28.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:28.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:28.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:28.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:28.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:28.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:29.279 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:31:29.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:29.280319+0000) 2019-10-18T04:31:30.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:30.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:30.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:30.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:31.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:31.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:31.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:31.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:32.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:32.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:32.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:32.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:32.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:32.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:33.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:33.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:33.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:33.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:33.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:31:33.427+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:33.431316+0000) 2019-10-18T04:31:33.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:31:33.911+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:33.917425+0000) 2019-10-18T04:31:34.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:34.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:34.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:34.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:34.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:31:34.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:34.744620+0000) 2019-10-18T04:31:35.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:35.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:35.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:35.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:36.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:36.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:36.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:36.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:37.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:37.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:37.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:37.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:37.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:37.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:38.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:38.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:38.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:38.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:39.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:39.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:39.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:39.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:39.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:31:39.274+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:39.280419+0000) 2019-10-18T04:31:40.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:40.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:40.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:40.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:41.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:41.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:41.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:41.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:42.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:42.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:42.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:42.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:42.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:42.490+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:43.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:43.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:43.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:43.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:43.430 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:31:43.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:43.431525+0000) 2019-10-18T04:31:43.916 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:31:43.910+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:43.917651+0000) 2019-10-18T04:31:44.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:44.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:44.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:44.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:44.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:31:44.738+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:44.744721+0000) 2019-10-18T04:31:45.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:45.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:45.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:45.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:46.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:46.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:46.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:46.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:47.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:47.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:47.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:47.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:47.498 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:47.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:48.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:48.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:48.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:48.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:49.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:49.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:49.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:49.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:49.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:31:49.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:49.280524+0000) 2019-10-18T04:31:50.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:50.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:50.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:50.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:51.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:51.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:51.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:51.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:52.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:52.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:52.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:52.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:52.498 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:52.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:53.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:53.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:53.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:53.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:53.431 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:31:53.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:53.431711+0000) 2019-10-18T04:31:53.917 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:31:53.910+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:53.917874+0000) 2019-10-18T04:31:54.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:54.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:54.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:54.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:54.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:31:54.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:54.744817+0000) 2019-10-18T04:31:55.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:55.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:55.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:55.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:56.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:56.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:56.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:56.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:57.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:57.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:57.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:57.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:57.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:31:57.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:31:58.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:58.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:58.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:58.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:59.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:59.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:31:59.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:31:59.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:31:59.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:31:59.273+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:31:59.280623+0000) 2019-10-18T04:32:00.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:00.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:00.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:00.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:01.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:01.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:01.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:01.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:02.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:02.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:02.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:02.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:02.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:02.493+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:03.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:03.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:03.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:03.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:03.431 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:32:03.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:03.431894+0000) 2019-10-18T04:32:03.917 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:32:03.910+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:03.918103+0000) 2019-10-18T04:32:04.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:04.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:04.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:04.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:04.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:32:04.737+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:04.744915+0000) 2019-10-18T04:32:05.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:05.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:05.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:05.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:06.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:06.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:06.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:06.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:07.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:07.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:07.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:07.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:07.502 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:07.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:08.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:08.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:08.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:08.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:09.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:09.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:09.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:09.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:09.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:32:09.277+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:09.280722+0000) 2019-10-18T04:32:10.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:10.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:10.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:11.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:11.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:11.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:11.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:12.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:12.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:12.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:12.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:12.502 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:12.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:13.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:13.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:13.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:13.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:13.431 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:32:13.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:13.432080+0000) 2019-10-18T04:32:13.918 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:32:13.914+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:13.918323+0000) 2019-10-18T04:32:14.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:14.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:14.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:14.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:14.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:32:14.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:14.745014+0000) 2019-10-18T04:32:15.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:15.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:15.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:15.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:16.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:16.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:16.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:16.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:17.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:17.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:17.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:17.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:17.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:17.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:18.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:18.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:18.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:18.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:19.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:19.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:19.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:19.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:19.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:32:19.277+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:19.280826+0000) 2019-10-18T04:32:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:20.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:20.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:21.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:21.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:21.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:21.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:22.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:22.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:22.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:22.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:22.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:22.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:23.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:23.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:23.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:23.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:23.431 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:32:23.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:23.432266+0000) 2019-10-18T04:32:23.917 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:32:23.914+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:23.918530+0000) 2019-10-18T04:32:24.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:24.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:24.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:24.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:24.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:32:24.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:24.745110+0000) 2019-10-18T04:32:25.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:25.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:25.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:25.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:26.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:26.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:26.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:26.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:27.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:27.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:27.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:27.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:27.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:27.500+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:28.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:28.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:28.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:28.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:29.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:29.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:29.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:29.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:29.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:32:29.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:29.280925+0000) 2019-10-18T04:32:30.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:30.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:30.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:30.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:31.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:31.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:31.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:31.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:32.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:32.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:32.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:32.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:32.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:32.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:33.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:33.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:33.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:33.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:33.432 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:32:33.426+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:33.432454+0000) 2019-10-18T04:32:33.918 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:32:33.914+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:33.918767+0000) 2019-10-18T04:32:34.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:34.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:34.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:34.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:34.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:32:34.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:34.745206+0000) 2019-10-18T04:32:35.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:35.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:35.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:35.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:36.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:36.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:36.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:36.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:37.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:37.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:37.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:37.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:37.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:37.504+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:38.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:38.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:38.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:38.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:39.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:39.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:39.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:39.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:39.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:32:39.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:39.281026+0000) 2019-10-18T04:32:40.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:40.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:40.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:40.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:41.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:41.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:41.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:41.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:42.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:42.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:42.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:42.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:42.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:42.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:43.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:43.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:43.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:43.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:43.432 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:32:43.430+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:43.432638+0000) 2019-10-18T04:32:43.918 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:32:43.918+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:43.919047+0000) 2019-10-18T04:32:44.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:44.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:44.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:44.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:44.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:32:44.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:44.745302+0000) 2019-10-18T04:32:45.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:45.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:45.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:45.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:46.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:46.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:46.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:46.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:47.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:47.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:47.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:47.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:47.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:47.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:48.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:48.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:48.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:48.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:48.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:48.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:48.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:48.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:49.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:32:49.276+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:49.281131+0000) 2019-10-18T04:32:50.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:50.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:50.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:50.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:51.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:51.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:51.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:52.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:52.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:52.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:52.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:52.510 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:52.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:52.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:52.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:52.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:52.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:53.432 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:32:53.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:53.432828+0000) 2019-10-18T04:32:53.920 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:32:53.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:53.919304+0000) 2019-10-18T04:32:54.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:54.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:54.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:54.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:54.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:32:54.740+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:54.745404+0000) 2019-10-18T04:32:55.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:55.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:55.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:55.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:56.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:56.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:56.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:56.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:56.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:56.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:56.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:56.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:57.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:32:57.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:32:57.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:57.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:57.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:57.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:58.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:58.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:58.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:58.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:32:59.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:32:59.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:32:59.281229+0000) 2019-10-18T04:32:59.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:59.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:32:59.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:32:59.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:00.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:00.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:00.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:00.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:01.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:01.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:01.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:01.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:02.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:02.507+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:03.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:03.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:03.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:03.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:03.432 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:33:03.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:03.433019+0000) 2019-10-18T04:33:03.918 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:33:03.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:03.919552+0000) 2019-10-18T04:33:04.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:04.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:04.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:04.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:04.744 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:33:04.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:04.745465+0000) 2019-10-18T04:33:04.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:04.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:04.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:04.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:05.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:05.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:05.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:05.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:06.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:06.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:06.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:06.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:07.513 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:07.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:07.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:07.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:07.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:07.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:08.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:08.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:08.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:08.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:09.280 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:09.275+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:09.281336+0000) 2019-10-18T04:33:09.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:09.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:09.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:09.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:10.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:10.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:10.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:10.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:11.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:11.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:11.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:11.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:12.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:12.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:12.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:12.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:12.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:12.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:13.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:33:13.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:13.433214+0000) 2019-10-18T04:33:13.919 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:33:13.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:13.919773+0000) 2019-10-18T04:33:13.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:13.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:13.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:13.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:14.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:33:14.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:14.745561+0000) 2019-10-18T04:33:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:14.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:14.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:14.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:15.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:15.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:15.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:15.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:16.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:16.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:16.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:16.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:17.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:17.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:17.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:17.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:17.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:17.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:18.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:18.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:18.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:18.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:19.281 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:19.279+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:19.281440+0000) 2019-10-18T04:33:19.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:19.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:19.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:19.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:20.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:20.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:20.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:20.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:21.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:21.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:21.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:21.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:22.516 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:22.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:22.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:22.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:22.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:22.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:23.432 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:33:23.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:23.433397+0000) 2019-10-18T04:33:23.919 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:33:23.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:23.920000+0000) 2019-10-18T04:33:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:23.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:23.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:24.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:33:24.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:24.745700+0000) 2019-10-18T04:33:24.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:24.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:24.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:24.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:25.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:25.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:25.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:25.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:26.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:26.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:26.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:26.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:27.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:27.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:27.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:27.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:27.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:27.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:28.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:28.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:28.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:28.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:29.285 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:29.279+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:29.281545+0000) 2019-10-18T04:33:29.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:29.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:29.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:29.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:30.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:30.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:30.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:30.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:31.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:31.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:31.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:31.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:32.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:32.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:32.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:32.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:32.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:32.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:33.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:33:33.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:33.433597+0000) 2019-10-18T04:33:33.919 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:33:33.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:33.920215+0000) 2019-10-18T04:33:33.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:33.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:33.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:33.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:34.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:33:34.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:34.745796+0000) 2019-10-18T04:33:35.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:34.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:35.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:34.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:36.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:36.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:36.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:36.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:36.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:36.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:36.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:36.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:37.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:37.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:37.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:37.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:37.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:37.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:38.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:38.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:38.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:38.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:39.281 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:39.278+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:39.281645+0000) 2019-10-18T04:33:39.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:39.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:39.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:39.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:40.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:40.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:40.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:40.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:42.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:42.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:42.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:42.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:42.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:42.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:43.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:43.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:43.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:43.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:43.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:33:43.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:43.433741+0000) 2019-10-18T04:33:43.919 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:33:43.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:43.920430+0000) 2019-10-18T04:33:44.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:44.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:44.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:44.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:44.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:33:44.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:44.745897+0000) 2019-10-18T04:33:44.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:44.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:44.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:44.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:45.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:45.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:45.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:45.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:47.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:47.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:47.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:47.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:47.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:47.514+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:48.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:47.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:48.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:47.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:48.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:48.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:48.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:48.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:49.281 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:49.278+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:49.281743+0000) 2019-10-18T04:33:50.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:50.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:50.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:50.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:51.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:51.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:51.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:51.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:52.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:52.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:52.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:52.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:52.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:52.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:53.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:53.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:53.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:53.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:53.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:33:53.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:53.433923+0000) 2019-10-18T04:33:53.920 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:33:53.917+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:53.920647+0000) 2019-10-18T04:33:54.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:54.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:54.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:54.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:54.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:33:54.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:54.745992+0000) 2019-10-18T04:33:55.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:55.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:55.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:55.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:56.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:56.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:56.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:56.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:57.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:57.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:57.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:57.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:57.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:33:57.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:33:58.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:58.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:58.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:58.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:59.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:59.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:33:59.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:33:59.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:33:59.281 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:59.278+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:59.281834+0000) 2019-10-18T04:33:59.282 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:33:59.278+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:33:59.283434+0000) 2019-10-18T04:34:00.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:00.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:00.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:00.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:01.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:01.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:01.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:01.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:02.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:02.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:02.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:02.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:02.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:02.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:03.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:03.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:03.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:03.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:03.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:34:03.429+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:03.434103+0000) 2019-10-18T04:34:03.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:34:03.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:03.782033+0000) 2019-10-18T04:34:03.920 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:34:03.916+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:03.920860+0000) 2019-10-18T04:34:04.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:04.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:04.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:04.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:04.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:34:04.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:04.746089+0000) 2019-10-18T04:34:05.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:05.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:05.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:05.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:06.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:06.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:06.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:06.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:07.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:07.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:07.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:07.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:07.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:07.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:08.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:08.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:08.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:08.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:09.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:09.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:09.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:09.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:10.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:10.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:10.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:10.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:11.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:11.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:11.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:11.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:12.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:12.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:12.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:12.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:12.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:12.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:13.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:13.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:13.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:13.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:13.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:34:13.432+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:13.434275+0000) 2019-10-18T04:34:13.781 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:34:13.777+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:13.782140+0000) 2019-10-18T04:34:13.920 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:34:13.916+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:13.921075+0000) 2019-10-18T04:34:14.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:14.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:14.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:14.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:14.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:34:14.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:14.746190+0000) 2019-10-18T04:34:15.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:15.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:15.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:15.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:15.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:15.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:15.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:15.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:17.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:17.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:17.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:17.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:17.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:17.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:18.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:18.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:18.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:18.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:19.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:19.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:19.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:19.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:20.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:20.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:20.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:20.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:21.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:21.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:21.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:21.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:22.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:22.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:22.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:22.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:22.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:22.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:23.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:23.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:23.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:23.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:23.433 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:34:23.432+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:23.434448+0000) 2019-10-18T04:34:23.781 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:34:23.777+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:23.782240+0000) 2019-10-18T04:34:23.920 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:34:23.916+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:23.921288+0000) 2019-10-18T04:34:24.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:24.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:24.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:24.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:24.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:34:24.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:24.746280+0000) 2019-10-18T04:34:25.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:25.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:25.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:25.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:26.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:26.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:26.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:26.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:27.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:27.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:27.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:27.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:27.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:27.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:28.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:28.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:28.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:28.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:29.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:29.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:29.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:29.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:30.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:30.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:30.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:30.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:31.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:31.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:31.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:31.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:32.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:32.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:32.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:32.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:32.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:32.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:33.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:33.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:33.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:33.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:33.434 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:34:33.432+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:33.434627+0000) 2019-10-18T04:34:33.781 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:34:33.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:33.782337+0000) 2019-10-18T04:34:33.920 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:34:33.916+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:33.921521+0000) 2019-10-18T04:34:33.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:33.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:33.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:33.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:34.745 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:34:34.745+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:34.746374+0000) 2019-10-18T04:34:34.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:34.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:34.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:34.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:35.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:35.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:35.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:35.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:36.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:36.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:36.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:36.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:37.529 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:37.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:37.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:37.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:37.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:37.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:38.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:38.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:38.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:38.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:39.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:39.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:39.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:39.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:40.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:40.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:40.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:40.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:41.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:41.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:41.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:41.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:42.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:42.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:42.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:42.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:42.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:42.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:43.434 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:34:43.432+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:43.434811+0000) 2019-10-18T04:34:43.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:34:43.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:43.782442+0000) 2019-10-18T04:34:43.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:43.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:43.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:43.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:43.921 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:34:43.916+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:43.921753+0000) 2019-10-18T04:34:44.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:34:44.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:44.746475+0000) 2019-10-18T04:34:44.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:44.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:44.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:44.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:45.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:45.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:45.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:45.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:46.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:46.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:46.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:46.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:47.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:47.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:47.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:47.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:47.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:47.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:48.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:48.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:48.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:48.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:49.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:49.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:49.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:49.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:50.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:50.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:50.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:50.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:51.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:51.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:51.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:51.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:52.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:52.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:52.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:52.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:52.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:52.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:53.434 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:34:53.432+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:53.434989+0000) 2019-10-18T04:34:53.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:34:53.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:53.782541+0000) 2019-10-18T04:34:53.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:53.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:53.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:53.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:53.921 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:34:53.916+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:53.921970+0000) 2019-10-18T04:34:54.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:34:54.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:34:54.746580+0000) 2019-10-18T04:34:54.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:54.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:54.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:54.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:55.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:55.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:55.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:55.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:56.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:56.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:56.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:56.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:57.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:34:57.528+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:34:57.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:57.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:57.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:57.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:58.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:58.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:58.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:58.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:34:59.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:59.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:34:59.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:34:59.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:00.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:00.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:00.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:01.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:01.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:01.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:01.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:02.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:02.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:02.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:02.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:02.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:02.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:03.435 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:03.432+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:03.435164+0000) 2019-10-18T04:35:03.436 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:03.432+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:03.436133+0000) 2019-10-18T04:35:03.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:35:03.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:03.782640+0000) 2019-10-18T04:35:03.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:03.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:03.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:03.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:03.921 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:35:03.920+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:03.922230+0000) 2019-10-18T04:35:04.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:35:04.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:04.746670+0000) 2019-10-18T04:35:04.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:04.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:04.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:04.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:05.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:05.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:05.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:05.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:06.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:06.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:06.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:06.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:07.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:07.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:07.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:07.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:07.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:07.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:07.934 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:07.932+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:07.935584+0000) 2019-10-18T04:35:08.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:08.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:08.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:08.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:09.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:09.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:09.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:09.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:10.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:10.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:10.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:10.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:11.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:11.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:11.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:12.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:12.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:12.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:12.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:12.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:12.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:13.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:35:13.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:13.782739+0000) 2019-10-18T04:35:13.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:13.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:13.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:13.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:13.921 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:35:13.920+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:13.922437+0000) 2019-10-18T04:35:14.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:35:14.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:14.746766+0000) 2019-10-18T04:35:14.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:14.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:14.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:14.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:15.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:15.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:15.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:15.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:16.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:16.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:16.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:16.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:17.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:17.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:17.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:17.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:17.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:17.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:17.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:17.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:17.935775+0000) 2019-10-18T04:35:18.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:18.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:18.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:18.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:19.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:19.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:19.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:19.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:20.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:20.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:20.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:20.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:21.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:21.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:21.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:21.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:22.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:22.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:22.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:22.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:22.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:22.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:23.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:23.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:23.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:23.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:23.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:35:23.779+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:23.782836+0000) 2019-10-18T04:35:23.922 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:35:23.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:23.922667+0000) 2019-10-18T04:35:24.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:35:24.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:24.746878+0000) 2019-10-18T04:35:24.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:24.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:24.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:24.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:25.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:25.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:25.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:25.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:26.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:26.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:26.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:26.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:27.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:27.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:27.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:27.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:27.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:27.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:27.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:27.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:27.935961+0000) 2019-10-18T04:35:28.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:28.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:28.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:28.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:29.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:29.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:29.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:29.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:30.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:30.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:30.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:30.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:31.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:31.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:31.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:31.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:32.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:32.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:32.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:32.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:32.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:32.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:33.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:33.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:33.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:33.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:33.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:35:33.779+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:33.782938+0000) 2019-10-18T04:35:33.922 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:35:33.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:33.922888+0000) 2019-10-18T04:35:34.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:35:34.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:34.746971+0000) 2019-10-18T04:35:34.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:34.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:34.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:34.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:35.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:35.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:35.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:35.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:36.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:36.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:36.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:36.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:37.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:37.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:37.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:37.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:37.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:37.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:37.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:37.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:37.936147+0000) 2019-10-18T04:35:38.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:38.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:38.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:38.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:39.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:39.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:39.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:39.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:40.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:40.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:40.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:40.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:41.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:41.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:41.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:41.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:42.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:42.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:42.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:42.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:42.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:42.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:43.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:43.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:43.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:43.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:43.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:35:43.779+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:43.783044+0000) 2019-10-18T04:35:43.922 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:35:43.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:43.923112+0000) 2019-10-18T04:35:44.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:44.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:44.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:44.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:44.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:35:44.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:44.747066+0000) 2019-10-18T04:35:45.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:45.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:45.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:45.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:46.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:46.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:46.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:46.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:47.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:47.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:47.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:47.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:25:10.669817+0000 (oldest deadline 2019-10-18T04:25:30.669817+0000) 2019-10-18T04:35:47.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:47.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:47.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:47.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:47.936317+0000) 2019-10-18T04:35:48.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:48.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:49.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:49.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:50.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:50.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:51.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:51.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:52.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:52.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:52.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:52.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:53.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:35:53.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:53.783140+0000) 2019-10-18T04:35:53.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:53.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:53.922 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:35:53.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:53.923354+0000) 2019-10-18T04:35:54.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:35:54.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:54.747163+0000) 2019-10-18T04:35:54.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:54.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:55.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:55.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:56.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:56.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:57.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:35:57.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:35:57.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:57.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:57.935 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:35:57.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:35:57.936498+0000) 2019-10-18T04:35:58.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:58.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:35:59.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:35:59.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:00.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:00.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:01.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:01.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:02.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:02.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:02.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:02.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:03.782 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:36:03.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:03.783245+0000) 2019-10-18T04:36:03.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:03.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:03.923 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:36:03.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:03.923598+0000) 2019-10-18T04:36:04.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:36:04.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:04.747262+0000) 2019-10-18T04:36:04.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:04.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:05.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:05.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:06.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:06.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:07.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:07.538+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:07.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:07.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:07.936 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:36:07.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:07.936681+0000) 2019-10-18T04:36:08.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:08.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:09.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:09.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:10.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:10.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:11.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:11.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:12.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:12.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:12.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:12.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:13.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:36:13.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:13.783346+0000) 2019-10-18T04:36:13.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:13.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:13.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:13.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:13.923 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:36:13.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:13.923790+0000) 2019-10-18T04:36:14.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:36:14.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:14.747359+0000) 2019-10-18T04:36:14.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:14.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:14.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:14.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:15.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:15.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:15.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:15.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:16.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:16.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:17.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:17.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:17.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:17.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:17.936 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:36:17.931+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:17.936868+0000) 2019-10-18T04:36:18.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:18.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:18.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:18.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:19.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:19.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:19.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:19.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:20.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:20.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:20.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:21.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:21.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:21.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:21.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:22.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:22.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:22.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:22.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:22.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:22.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:23.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:23.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:23.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:23.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:23.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:36:23.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:23.783450+0000) 2019-10-18T04:36:23.924 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:36:23.919+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:23.924025+0000) 2019-10-18T04:36:24.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:24.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:24.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:24.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:24.746 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:36:24.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:24.747458+0000) 2019-10-18T04:36:25.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:25.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:25.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:25.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:26.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:26.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:27.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:27.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:27.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:27.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:27.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:27.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:27.936 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:36:27.930+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:27.937050+0000) 2019-10-18T04:36:28.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:28.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:28.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:28.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:29.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:29.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:29.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:29.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:30.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:30.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:30.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:30.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:31.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:31.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:32.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:32.541+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:32.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:32.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:32.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:32.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:33.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:36:33.777+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:33.783558+0000) 2019-10-18T04:36:33.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:33.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:33.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:33.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:33.923 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:36:33.918+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:33.924262+0000) 2019-10-18T04:36:34.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:36:34.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:34.747556+0000) 2019-10-18T04:36:34.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:34.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:34.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:34.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:35.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:35.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:35.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:35.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:36.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:36.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:36.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:36.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:37.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:37.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:37.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:37.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:37.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:37.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:37.936 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:36:37.930+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:37.937242+0000) 2019-10-18T04:36:38.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:38.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:38.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:38.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:39.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:39.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:39.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:39.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:40.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:40.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:40.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:40.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:41.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:41.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:41.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:41.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:42.551 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:42.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:42.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:42.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:42.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:42.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:43.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:36:43.777+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:43.783663+0000) 2019-10-18T04:36:43.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:43.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:43.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:43.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:43.924 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:36:43.918+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:43.924475+0000) 2019-10-18T04:36:44.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:36:44.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:44.747657+0000) 2019-10-18T04:36:44.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:44.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:44.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:44.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:45.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:45.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:45.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:45.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:46.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:46.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:46.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:46.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:47.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:47.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:47.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:47.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:47.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:47.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:47.937 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:36:47.930+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:47.937453+0000) 2019-10-18T04:36:48.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:48.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:48.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:48.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:49.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:49.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:49.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:49.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:50.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:50.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:50.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:50.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:51.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:51.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:51.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:51.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:52.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:52.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:52.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:52.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:52.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:52.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:53.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:36:53.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:53.783765+0000) 2019-10-18T04:36:53.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:53.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:53.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:53.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:53.924 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:36:53.918+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:53.924716+0000) 2019-10-18T04:36:54.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:36:54.745+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:54.747759+0000) 2019-10-18T04:36:54.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:54.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:54.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:54.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:55.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:55.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:55.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:55.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:56.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:56.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:56.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:56.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:57.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:36:57.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:36:57.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:57.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:57.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:57.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:57.937 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:36:57.930+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:36:57.937649+0000) 2019-10-18T04:36:58.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:58.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:58.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:58.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:36:59.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:59.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:36:59.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:36:59.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:00.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:00.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:00.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:00.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:01.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:01.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:01.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:01.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:02.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:02.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:02.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:02.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:02.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:02.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:03.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:03.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:03.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:03.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:03.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:37:03.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:03.783868+0000) 2019-10-18T04:37:03.924 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:37:03.918+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:03.924941+0000) 2019-10-18T04:37:04.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:04.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:04.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:04.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:04.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:37:04.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:04.747858+0000) 2019-10-18T04:37:05.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:05.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:05.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:05.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:06.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:06.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:06.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:06.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:07.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:07.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:07.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:07.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:07.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:07.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:07.937 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:37:07.930+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:07.937832+0000) 2019-10-18T04:37:08.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:08.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:08.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:08.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:09.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:09.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:09.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:09.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:10.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:10.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:10.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:10.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:11.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:11.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:11.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:11.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:12.556 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:12.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:12.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:12.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:12.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:12.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:13.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:13.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:13.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:13.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:13.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:37:13.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:13.783967+0000) 2019-10-18T04:37:13.924 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:37:13.918+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:13.925158+0000) 2019-10-18T04:37:14.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:14.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:14.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:14.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:14.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:37:14.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:14.747953+0000) 2019-10-18T04:37:15.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:15.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:15.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:15.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:16.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:16.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:16.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:17.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:17.552+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:17.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:17.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:17.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:17.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:17.937 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:37:17.930+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:17.938011+0000) 2019-10-18T04:37:18.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:18.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:18.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:18.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:19.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:19.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:19.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:19.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:20.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:20.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:20.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:20.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:21.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:21.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:21.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:21.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:22.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:22.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:22.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:22.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:22.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:22.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:23.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:23.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:23.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:23.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:23.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:37:23.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:23.784069+0000) 2019-10-18T04:37:23.924 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:37:23.922+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:23.925402+0000) 2019-10-18T04:37:24.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:37:24.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:24.748046+0000) 2019-10-18T04:37:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:24.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:24.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:25.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:25.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:25.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:25.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:26.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:26.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:26.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:26.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:27.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:27.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:27.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:27.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:27.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:27.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:27.937 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:37:27.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:27.938207+0000) 2019-10-18T04:37:28.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:28.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:28.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:28.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:29.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:29.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:29.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:29.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:30.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:30.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:30.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:30.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:31.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:31.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:31.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:31.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:32.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:32.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:32.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:32.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:32.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:32.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:33.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:33.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:33.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:33.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:33.783 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:37:33.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:33.784171+0000) 2019-10-18T04:37:33.925 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:37:33.922+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:33.925623+0000) 2019-10-18T04:37:34.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:34.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:34.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:34.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:34.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:37:34.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:34.748144+0000) 2019-10-18T04:37:35.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:35.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:35.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:35.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:36.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:36.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:36.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:36.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:37.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:37.555+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:37.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:37.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:37.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:37.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:37.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:37:37.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:37.938397+0000) 2019-10-18T04:37:38.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:38.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:38.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:38.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:39.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:39.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:39.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:39.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:40.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:40.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:40.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:40.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:41.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:41.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:41.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:41.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:42.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:42.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:42.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:42.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:42.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:42.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:43.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:43.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:43.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:43.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:43.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:37:43.779+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:43.784283+0000) 2019-10-18T04:37:43.925 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:37:43.921+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:43.925869+0000) 2019-10-18T04:37:44.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:44.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:44.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:44.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:44.747 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:37:44.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:44.748243+0000) 2019-10-18T04:37:45.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:45.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:45.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:45.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:46.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:46.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:46.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:46.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:47.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:47.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:47.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:47.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:47.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:47.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:47.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:37:47.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:47.938541+0000) 2019-10-18T04:37:48.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:48.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:48.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:48.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:49.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:49.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:49.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:49.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:50.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:50.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:50.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:50.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:51.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:51.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:51.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:51.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:52.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:52.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:52.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:52.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:52.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:52.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:53.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:53.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:53.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:53.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:53.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:37:53.779+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:53.784389+0000) 2019-10-18T04:37:53.925 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:37:53.921+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:53.926086+0000) 2019-10-18T04:37:54.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:54.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:54.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:54.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:54.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:37:54.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:54.748336+0000) 2019-10-18T04:37:55.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:55.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:55.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:55.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:56.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:56.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:56.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:56.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:57.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:37:57.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:37:57.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:57.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:57.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:57.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:57.939 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:37:57.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:37:57.938750+0000) 2019-10-18T04:37:58.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:58.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:58.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:58.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:37:59.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:59.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:37:59.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:37:59.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:00.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:00.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:00.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:00.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:01.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:01.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:01.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:01.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:02.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:02.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:02.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:02.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:02.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:02.559+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:03.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:03.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:03.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:03.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:03.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:38:03.779+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:03.784492+0000) 2019-10-18T04:38:03.925 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:38:03.925+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:03.926320+0000) 2019-10-18T04:38:04.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:04.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:04.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:04.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:04.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:38:04.743+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:04.748433+0000) 2019-10-18T04:38:05.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:05.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:05.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:05.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:06.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:06.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:06.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:06.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:07.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:07.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:07.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:07.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:07.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:07.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:07.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:38:07.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:07.938962+0000) 2019-10-18T04:38:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:08.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:08.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:08.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:09.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:09.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:09.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:09.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:10.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:10.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:10.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:10.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:11.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:11.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:11.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:11.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:12.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:12.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:12.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:12.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:12.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:12.562+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:13.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:13.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:13.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:13.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:13.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:38:13.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:13.784592+0000) 2019-10-18T04:38:13.926 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:38:13.925+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:13.926542+0000) 2019-10-18T04:38:14.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:14.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:14.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:14.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:14.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:38:14.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:14.748529+0000) 2019-10-18T04:38:15.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:15.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:15.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:15.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:16.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:16.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:16.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:16.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:17.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:17.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:17.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:17.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:17.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:17.562+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:17.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:38:17.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:17.939144+0000) 2019-10-18T04:38:18.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:18.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:18.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:18.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:19.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:19.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:19.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:19.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:20.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:20.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:20.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:20.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:21.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:21.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:21.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:21.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:22.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:22.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:22.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:22.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:22.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:22.562+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:23.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:23.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:23.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:23.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:23.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:38:23.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:23.784695+0000) 2019-10-18T04:38:23.926 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:38:23.925+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:23.926756+0000) 2019-10-18T04:38:24.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:24.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:24.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:24.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:24.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:38:24.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:24.748623+0000) 2019-10-18T04:38:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:25.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:25.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:25.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:26.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:26.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:26.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:26.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:27.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:27.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:27.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:27.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:27.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:38:27.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:27.939326+0000) 2019-10-18T04:38:28.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:28.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:28.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:28.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:29.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:29.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:29.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:29.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:30.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:30.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:30.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:30.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:31.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:31.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:31.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:31.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:32.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:32.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:32.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:32.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:32.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:32.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:33.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:33.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:33.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:33.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:33.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:38:33.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:33.784803+0000) 2019-10-18T04:38:33.926 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:38:33.925+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:33.926972+0000) 2019-10-18T04:38:34.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:34.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:34.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:34.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:34.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:38:34.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:34.748722+0000) 2019-10-18T04:38:35.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:35.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:35.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:35.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:36.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:36.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:36.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:36.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:37.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:37.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:37.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:37.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:37.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:37.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:37.938 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:38:37.937+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:37.939508+0000) 2019-10-18T04:38:38.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:38.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:38.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:38.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:39.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:39.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:39.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:39.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:40.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:40.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:40.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:40.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:41.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:41.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:41.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:41.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:42.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:42.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:42.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:42.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:42.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:42.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:43.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:43.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:43.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:43.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:43.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:38:43.778+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:43.784907+0000) 2019-10-18T04:38:43.927 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:38:43.925+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:43.927209+0000) 2019-10-18T04:38:44.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:44.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:44.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:44.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:44.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:38:44.742+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:44.748821+0000) 2019-10-18T04:38:45.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:45.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:45.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:45.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:46.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:46.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:46.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:46.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:47.572 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:47.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:47.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:47.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:47.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:47.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:47.939 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:38:47.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:47.939665+0000) 2019-10-18T04:38:48.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:48.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:48.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:48.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:49.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:49.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:49.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:49.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:50.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:50.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:50.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:50.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:51.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:51.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:51.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:51.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:52.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:52.565+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:52.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:52.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:52.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:52.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:53.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:53.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:53.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:53.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:53.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:38:53.777+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:53.785013+0000) 2019-10-18T04:38:53.926 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:38:53.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:53.927450+0000) 2019-10-18T04:38:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:54.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:54.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:54.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:38:54.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:54.748913+0000) 2019-10-18T04:38:55.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:55.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:55.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:55.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:56.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:56.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:56.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:56.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:57.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:38:57.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:38:57.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:57.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:57.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:57.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:57.940 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:38:57.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:38:57.939854+0000) 2019-10-18T04:38:58.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:58.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:58.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:58.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:38:59.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:59.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:38:59.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:38:59.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:00.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:00.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:00.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:01.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:01.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:01.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:01.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:02.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:02.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:02.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:02.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:02.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:02.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:03.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:03.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:03.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:03.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:03.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:39:03.777+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:03.785118+0000) 2019-10-18T04:39:03.927 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:39:03.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:03.927682+0000) 2019-10-18T04:39:04.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:04.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:04.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:04.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:04.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:39:04.741+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:04.749007+0000) 2019-10-18T04:39:05.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:05.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:05.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:05.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:06.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:06.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:06.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:06.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:07.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:07.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:07.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:07.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:07.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:07.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:07.939 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:39:07.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:07.940041+0000) 2019-10-18T04:39:08.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:08.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:08.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:08.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:09.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:09.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:09.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:09.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:10.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:10.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:10.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:10.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:11.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:11.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:11.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:11.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:12.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:12.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:12.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:12.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:12.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:12.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:13.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:13.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:13.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:13.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:13.784 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:39:13.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:13.785226+0000) 2019-10-18T04:39:13.927 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:39:13.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:13.927916+0000) 2019-10-18T04:39:14.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:14.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:14.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:14.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:14.748 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:39:14.745+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:14.749106+0000) 2019-10-18T04:39:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:15.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:15.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:15.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:16.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:16.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:16.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:16.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:17.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:17.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:17.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:17.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:17.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:17.577+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:17.939 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:39:17.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:17.940252+0000) 2019-10-18T04:39:18.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:18.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:18.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:18.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:19.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:19.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:19.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:19.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:20.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:20.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:20.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:20.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:21.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:21.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:21.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:22.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:22.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:22.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:22.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:22.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:22.576+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:23.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:23.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:23.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:23.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:23.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:39:23.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:23.785328+0000) 2019-10-18T04:39:23.928 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:39:23.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:23.928174+0000) 2019-10-18T04:39:24.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:24.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:24.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:24.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:24.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:39:24.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:24.749199+0000) 2019-10-18T04:39:25.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:25.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:25.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:25.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:26.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:26.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:26.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:26.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:27.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:27.576+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:27.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:27.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:27.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:27.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:27.939 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:39:27.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:27.940433+0000) 2019-10-18T04:39:28.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:28.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:28.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:28.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:29.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:29.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:29.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:29.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:30.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:30.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:30.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:30.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:31.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:31.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:31.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:31.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:32.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:32.576+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:32.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:32.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:32.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:32.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:33.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:33.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:33.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:33.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:33.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:39:33.780+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:33.785428+0000) 2019-10-18T04:39:33.927 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:39:33.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:33.928396+0000) 2019-10-18T04:39:34.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:34.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:34.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:34.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:34.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:39:34.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:34.749296+0000) 2019-10-18T04:39:35.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:35.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:35.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:35.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:36.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:36.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:36.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:36.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:37.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:37.576+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:37.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:37.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:37.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:37.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:37.941 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:39:37.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:37.940615+0000) 2019-10-18T04:39:38.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:38.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:38.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:38.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:39.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:39.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:39.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:39.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:40.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:40.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:40.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:40.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:41.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:41.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:41.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:41.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:42.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:42.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:42.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:42.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:42.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:42.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:43.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:43.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:43.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:43.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:43.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:39:43.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:43.785540+0000) 2019-10-18T04:39:43.928 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:39:43.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:43.928639+0000) 2019-10-18T04:39:44.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:44.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:44.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:44.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:44.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:39:44.744+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:44.749394+0000) 2019-10-18T04:39:45.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:45.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:45.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:45.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:46.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:46.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:46.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:46.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:47.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:47.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:47.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:47.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:47.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:47.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:47.940 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:39:47.936+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:47.940759+0000) 2019-10-18T04:39:48.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:48.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:48.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:48.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:49.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:49.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:49.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:49.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:50.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:50.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:50.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:50.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:51.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:51.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:51.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:51.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:52.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:52.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:52.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:52.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:52.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:52.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:53.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:53.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:53.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:53.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:53.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:39:53.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:53.785646+0000) 2019-10-18T04:39:53.928 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:39:53.924+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:53.928862+0000) 2019-10-18T04:39:54.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:39:54.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:54.749488+0000) 2019-10-18T04:39:54.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:54.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:54.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:54.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:55.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:55.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:55.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:55.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:56.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:56.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:56.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:56.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:57.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:39:57.579+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:39:57.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:57.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:57.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:57.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:57.940 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:39:57.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:39:57.940956+0000) 2019-10-18T04:39:58.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:58.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:58.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:58.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:39:59.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:59.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:39:59.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:39:59.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:00.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:00.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:00.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:00.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:01.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:01.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:01.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:01.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:02.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:02.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:02.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:02.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:02.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:02.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:03.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:40:03.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:03.785747+0000) 2019-10-18T04:40:03.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:03.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:03.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:03.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:03.928 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:40:03.923+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:03.929096+0000) 2019-10-18T04:40:04.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:04.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:04.749599+0000) 2019-10-18T04:40:04.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:04.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:04.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:04.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:05.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:05.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:05.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:05.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:06.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:06.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:06.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:06.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:07.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:07.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:07.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:07.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:07.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:07.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:07.940 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:40:07.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:07.941169+0000) 2019-10-18T04:40:08.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:08.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:08.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:08.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:09.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:09.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:09.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:09.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:10.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:10.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:10.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:10.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:11.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:11.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:11.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:11.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:12.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:12.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:12.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:12.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:12.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:12.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:13.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:40:13.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:13.785846+0000) 2019-10-18T04:40:13.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:13.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:13.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:13.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:13.928 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:40:13.923+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:13.929322+0000) 2019-10-18T04:40:14.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:14.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:14.749688+0000) 2019-10-18T04:40:14.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:14.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:14.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:14.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:15.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:15.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:15.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:15.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:16.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:16.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:16.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:16.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:17.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:17.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:17.940 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:40:17.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:17.941348+0000) 2019-10-18T04:40:17.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:17.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:17.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:17.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:18.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:18.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:18.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:18.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:20.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:20.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:20.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:20.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:21.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:20.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:21.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:20.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:22.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:22.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:22.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:22.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:22.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:22.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:23.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:23.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:23.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:23.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:23.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:40:23.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:23.785950+0000) 2019-10-18T04:40:23.929 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:40:23.923+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:23.929555+0000) 2019-10-18T04:40:24.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:24.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:24.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:24.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:24.749 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:24.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:24.749779+0000) 2019-10-18T04:40:24.750 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:24.747+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:24.751069+0000) 2019-10-18T04:40:25.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:25.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:25.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:25.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:26.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:26.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:26.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:26.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:27.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:27.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:27.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:27.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:27.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:27.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:27.943 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:40:27.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:27.941528+0000) 2019-10-18T04:40:28.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:28.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:28.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:28.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:29.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:29.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:29.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:29.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:29.250 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:29.247+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:29.249972+0000) 2019-10-18T04:40:30.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:30.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:30.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:30.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:31.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:31.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:31.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:31.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:32.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:32.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:32.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:32.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:32.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:32.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:33.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:33.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:33.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:33.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:33.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:40:33.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:33.786059+0000) 2019-10-18T04:40:33.929 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:40:33.923+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:33.929772+0000) 2019-10-18T04:40:34.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:34.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:34.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:34.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:35.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:35.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:35.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:35.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:36.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:36.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:36.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:36.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:37.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:37.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:37.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:37.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:37.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:37.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:37.941 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:40:37.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:37.941708+0000) 2019-10-18T04:40:38.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:38.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:38.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:38.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:39.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:39.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:39.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:39.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:39.250 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:39.246+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:39.250072+0000) 2019-10-18T04:40:40.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:40.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:40.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:40.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:41.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:41.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:41.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:41.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:42.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:42.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:42.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:42.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:43.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:43.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:43.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:43.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:43.785 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:40:43.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:43.786160+0000) 2019-10-18T04:40:43.929 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:40:43.923+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:43.930026+0000) 2019-10-18T04:40:44.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:44.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:44.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:44.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:45.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:45.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:45.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:45.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:46.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:46.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:46.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:46.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:47.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:47.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:47.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:47.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:47.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:47.586+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:47.941 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:40:47.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:47.941892+0000) 2019-10-18T04:40:48.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:48.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:48.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:48.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:49.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:49.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:49.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:49.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:49.249 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:49.246+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:49.250178+0000) 2019-10-18T04:40:50.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:50.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:50.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:50.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:51.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:51.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:51.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:51.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:52.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:52.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:52.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:52.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:52.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:52.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:53.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:53.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:53.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:53.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:53.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:40:53.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:53.786258+0000) 2019-10-18T04:40:53.929 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:40:53.927+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:53.930288+0000) 2019-10-18T04:40:54.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:54.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:54.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:54.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:55.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:55.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:55.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:55.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:56.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:56.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:56.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:56.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:57.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:57.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:57.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:57.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:57.595 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:40:57.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:40:57.941 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:40:57.935+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:57.942075+0000) 2019-10-18T04:40:58.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:58.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:58.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:58.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:59.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:59.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:40:59.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:40:59.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:40:59.250 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:40:59.246+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:40:59.250285+0000) 2019-10-18T04:41:00.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:00.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:00.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:00.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:01.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:01.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:01.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:01.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:02.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:02.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:02.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:02.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:02.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:02.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:03.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:03.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:03.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:03.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:03.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:41:03.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:03.786365+0000) 2019-10-18T04:41:03.929 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:41:03.927+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:03.930540+0000) 2019-10-18T04:41:04.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:04.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:04.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:04.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:05.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:05.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:05.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:05.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:06.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:06.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:06.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:06.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:07.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:07.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:07.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:07.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:07.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:07.589+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:07.941 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:41:07.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:07.942258+0000) 2019-10-18T04:41:08.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:08.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:08.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:08.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:09.250 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:09.245+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:09.250387+0000) 2019-10-18T04:41:09.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:09.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:09.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:09.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:10.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:10.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:10.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:10.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:11.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:11.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:11.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:11.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:12.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:12.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:12.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:12.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:12.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:12.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:13.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:13.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:13.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:13.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:13.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:41:13.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:13.786468+0000) 2019-10-18T04:41:13.930 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:41:13.926+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:13.930765+0000) 2019-10-18T04:41:14.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:14.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:14.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:14.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:15.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:15.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:15.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:15.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:16.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:16.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:16.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:16.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:17.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:17.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:17.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:17.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:17.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:17.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:17.941 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:41:17.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:17.942437+0000) 2019-10-18T04:41:18.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:18.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:18.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:18.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:19.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:19.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:19.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:19.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:19.251 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:19.245+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:19.250493+0000) 2019-10-18T04:41:20.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:20.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:20.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:20.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:21.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:21.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:21.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:21.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:22.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:22.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:22.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:22.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:23.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:23.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:23.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:23.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:23.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:41:23.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:23.786569+0000) 2019-10-18T04:41:23.930 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:41:23.926+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:23.930993+0000) 2019-10-18T04:41:24.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:24.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:24.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:24.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:25.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:25.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:25.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:25.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:26.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:26.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:26.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:26.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:27.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:27.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:27.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:27.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:27.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:27.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:27.942 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:41:27.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:27.942621+0000) 2019-10-18T04:41:28.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:28.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:28.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:28.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:29.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:29.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:29.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:29.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:29.252 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:29.249+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:29.250593+0000) 2019-10-18T04:41:30.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:30.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:30.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:30.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:31.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:31.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:31.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:31.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:32.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:32.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:32.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:32.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:32.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:32.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:33.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:33.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:33.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:33.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:33.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:41:33.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:33.786670+0000) 2019-10-18T04:41:33.930 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:41:33.926+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:33.931209+0000) 2019-10-18T04:41:33.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:33.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:33.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:33.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:35.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:35.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:35.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:35.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:36.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:36.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:36.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:36.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:37.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:37.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:37.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:37.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:37.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:37.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:37.942 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:41:37.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:37.942802+0000) 2019-10-18T04:41:37.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:37.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:37.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:37.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:38.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:38.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:38.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:38.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:39.250 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:39.249+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:39.250691+0000) 2019-10-18T04:41:39.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:39.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:39.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:39.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:40.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:40.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:40.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:40.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:41.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:41.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:41.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:41.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:42.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:42.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:42.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:42.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:42.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:42.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:43.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:41:43.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:43.786773+0000) 2019-10-18T04:41:43.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:43.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:43.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:43.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:43.930 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:41:43.926+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:43.931436+0000) 2019-10-18T04:41:44.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:44.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:44.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:44.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:45.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:45.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:45.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:45.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:46.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:46.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:46.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:46.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:47.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:47.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:47.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:47.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:47.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:47.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:47.942 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:41:47.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:47.942982+0000) 2019-10-18T04:41:48.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:48.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:48.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:48.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:49.250 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:49.248+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:49.250789+0000) 2019-10-18T04:41:49.251 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:49.248+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:49.251903+0000) 2019-10-18T04:41:49.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:49.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:49.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:49.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:50.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:50.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:50.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:50.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:51.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:51.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:51.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:51.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:52.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:52.600+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:52.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:52.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:52.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:52.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:53.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:53.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:53.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:53.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 420 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:53.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:41:53.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:53.751000+0000) 2019-10-18T04:41:53.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:41:53.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:53.786879+0000) 2019-10-18T04:41:53.931 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:41:53.926+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:53.931702+0000) 2019-10-18T04:41:54.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:54.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:54.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:54.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:55.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:55.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:56.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:56.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:56.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:56.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:57.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:41:57.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:41:57.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:57.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:57.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:57.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:57.942 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:41:57.938+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:41:57.943162+0000) 2019-10-18T04:41:58.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:58.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:58.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:58.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:41:59.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:59.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:41:59.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:41:59.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:00.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:00.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:00.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:00.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:01.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:01.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:01.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:01.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:02.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:02.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:02.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:02.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:02.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:02.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:03.750 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:42:03.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:03.751098+0000) 2019-10-18T04:42:03.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:42:03.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:03.786981+0000) 2019-10-18T04:42:03.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:03.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:03.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:03.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:03.931 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:42:03.926+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:03.931964+0000) 2019-10-18T04:42:04.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:04.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:04.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:04.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:05.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:05.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:05.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:05.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:06.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:06.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:06.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:06.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:07.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:07.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:07.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:07.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:07.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:07.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:07.942 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:42:07.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:07.943341+0000) 2019-10-18T04:42:08.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:08.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:08.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:08.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:09.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:09.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:09.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:09.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:10.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:10.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:10.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:10.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:11.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:11.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:11.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:11.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:12.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:12.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:12.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:12.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:12.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:12.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:13.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:13.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:13.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:13.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:13.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:42:13.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:13.751197+0000) 2019-10-18T04:42:13.786 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:42:13.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:13.787084+0000) 2019-10-18T04:42:13.933 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:42:13.930+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:13.932205+0000) 2019-10-18T04:42:14.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:14.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:14.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:14.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:15.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:15.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:15.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:15.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:16.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:16.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:16.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:16.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:17.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:17.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:17.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:17.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:17.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:17.603+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:17.942 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:42:17.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:17.943522+0000) 2019-10-18T04:42:18.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:18.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:18.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:18.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:19.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:19.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:19.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:19.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:20.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:20.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:20.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:20.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:21.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:21.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:21.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:21.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:22.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:22.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:22.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:22.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:22.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:22.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:23.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:23.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:23.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:23.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:23.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:42:23.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:23.751299+0000) 2019-10-18T04:42:23.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:42:23.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:23.787185+0000) 2019-10-18T04:42:23.931 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:42:23.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:23.932454+0000) 2019-10-18T04:42:24.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:24.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:24.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:24.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:25.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:25.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:25.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:25.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:26.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:26.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:26.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:26.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:27.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:27.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:27.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:27.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:27.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:27.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:27.943 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:42:27.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:27.943701+0000) 2019-10-18T04:42:28.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:28.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:28.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:28.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:29.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:29.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:29.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:29.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:30.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:30.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:30.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:30.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:31.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:31.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:31.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:31.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:32.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:32.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:32.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:32.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:32.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:32.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:33.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:33.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:33.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:33.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:33.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:42:33.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:33.751405+0000) 2019-10-18T04:42:33.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:42:33.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:33.787288+0000) 2019-10-18T04:42:33.939 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:42:33.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:33.932714+0000) 2019-10-18T04:42:34.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:34.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:34.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:34.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:35.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:35.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:35.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:35.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:36.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:36.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:36.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:36.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:37.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:37.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:37.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:37.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:37.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:37.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:37.943 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:42:37.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:37.943885+0000) 2019-10-18T04:42:38.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:38.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:38.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:38.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:39.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:39.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:39.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:39.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:40.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:40.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:40.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:40.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:41.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:41.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:41.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:41.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:42.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:42.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:42.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:42.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:42.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:42.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:43.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:43.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:43.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:43.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:43.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:42:43.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:43.751509+0000) 2019-10-18T04:42:43.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:42:43.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:43.787389+0000) 2019-10-18T04:42:43.932 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:42:43.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:43.932944+0000) 2019-10-18T04:42:44.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:44.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:44.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:44.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:45.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:45.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:45.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:45.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:46.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:46.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:46.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:46.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:47.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:47.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:47.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:47.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:47.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:47.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:47.943 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:42:47.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:47.944096+0000) 2019-10-18T04:42:48.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:48.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:48.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:48.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:49.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:49.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:49.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:49.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:50.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:50.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:50.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:50.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:51.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:51.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:51.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:51.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:52.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:52.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:52.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:52.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:52.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:52.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:53.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:53.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:53.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:53.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:53.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:42:53.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:53.751613+0000) 2019-10-18T04:42:53.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:42:53.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:53.787489+0000) 2019-10-18T04:42:53.932 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:42:53.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:53.933157+0000) 2019-10-18T04:42:54.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:54.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:54.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:54.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:55.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:55.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:55.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:55.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:56.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:56.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:56.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:56.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:57.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:57.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:57.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:57.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:57.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:42:57.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:42:57.943 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:42:57.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:42:57.944308+0000) 2019-10-18T04:42:58.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:58.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:58.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:58.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:42:59.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:59.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:42:59.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:42:59.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:00.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:00.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:00.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:00.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:01.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:01.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:01.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:01.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:02.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:02.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:02.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:02.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:02.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:02.610+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:03.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:03.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:03.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:03.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:03.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:43:03.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:03.751719+0000) 2019-10-18T04:43:03.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:43:03.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:03.787600+0000) 2019-10-18T04:43:03.932 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:43:03.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:03.933406+0000) 2019-10-18T04:43:04.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:04.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:04.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:04.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:05.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:05.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:05.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:05.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:06.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:06.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:06.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:06.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:07.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:07.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:07.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:07.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:07.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:07.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:07.943 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:43:07.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:07.944488+0000) 2019-10-18T04:43:08.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:08.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:08.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:08.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:09.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:09.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:09.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:09.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:10.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:10.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:10.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:10.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:11.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:11.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:11.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:11.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:12.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:12.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:12.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:12.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:12.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:12.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:13.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:13.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:13.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:13.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:13.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:43:13.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:13.751828+0000) 2019-10-18T04:43:13.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:43:13.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:13.787704+0000) 2019-10-18T04:43:13.933 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:43:13.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:13.933647+0000) 2019-10-18T04:43:14.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:14.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:14.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:14.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:15.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:15.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:15.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:15.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:16.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:16.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:16.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:16.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:17.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:17.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:17.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:17.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:17.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:17.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:17.944 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:43:17.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:17.944667+0000) 2019-10-18T04:43:18.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:18.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:18.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:18.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:19.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:19.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:19.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:19.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:20.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:20.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:20.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:20.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:21.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:21.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:21.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:21.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:22.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:22.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:22.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:22.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:22.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:22.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:23.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:23.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:23.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:23.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:23.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:43:23.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:23.751927+0000) 2019-10-18T04:43:23.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:43:23.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:23.787803+0000) 2019-10-18T04:43:23.933 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:43:23.929+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:23.933888+0000) 2019-10-18T04:43:24.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:24.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:24.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:24.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:25.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:25.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:25.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:25.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:26.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:26.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:26.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:26.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:27.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:27.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:27.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:27.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:27.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:27.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:27.944 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:43:27.941+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:27.944851+0000) 2019-10-18T04:43:28.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:28.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:28.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:28.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:29.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:29.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:29.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:29.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:30.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:30.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:30.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:30.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:31.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:31.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:31.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:31.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:32.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:32.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:32.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:32.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:32.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:32.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:33.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:33.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:33.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:33.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:33.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:43:33.745+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:33.752027+0000) 2019-10-18T04:43:33.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:43:33.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:33.787905+0000) 2019-10-18T04:43:33.933 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:43:33.928+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:33.934129+0000) 2019-10-18T04:43:34.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:34.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:34.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:34.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:35.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:35.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:35.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:35.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:36.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:36.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:36.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:36.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:37.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:37.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:37.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:37.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:37.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:37.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:37.944 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:43:37.940+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:37.945063+0000) 2019-10-18T04:43:38.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:38.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:38.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:38.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:39.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:39.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:39.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:39.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:40.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:40.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:40.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:40.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:41.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:41.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:41.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:41.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:42.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:42.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:42.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:42.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:42.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:42.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:43.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:43.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:43.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:43.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:43.751 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:43:43.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:43.752134+0000) 2019-10-18T04:43:43.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:43:43.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:43.788006+0000) 2019-10-18T04:43:43.933 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:43:43.932+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:43.934342+0000) 2019-10-18T04:43:44.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:44.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:44.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:44.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:45.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:45.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:45.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:45.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:46.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:46.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:46.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:46.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:47.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:47.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:47.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:47.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:47.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:47.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:47.944 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:43:47.940+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:47.945244+0000) 2019-10-18T04:43:48.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:48.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:48.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:48.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:49.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:49.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:49.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:49.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:50.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:50.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:50.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:50.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:51.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:51.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:51.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:51.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:52.626 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:52.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:52.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:52.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:52.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:52.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:53.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:53.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:53.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:53.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:53.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:43:53.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:53.752238+0000) 2019-10-18T04:43:53.787 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:43:53.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:53.788106+0000) 2019-10-18T04:43:53.934 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:43:53.932+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:53.934563+0000) 2019-10-18T04:43:54.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:54.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:54.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:54.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:55.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:55.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:55.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:55.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:56.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:56.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:56.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:56.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:57.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:57.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:57.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:57.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:57.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:43:57.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:43:57.944 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:43:57.940+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:43:57.945425+0000) 2019-10-18T04:43:58.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:58.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:58.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:58.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:43:59.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:59.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:43:59.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:43:59.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:00.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:00.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:00.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:00.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:01.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:01.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:01.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:01.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:02.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:02.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:02.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:02.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:02.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:02.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:03.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:03.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:03.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:03.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:03.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:44:03.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:03.752340+0000) 2019-10-18T04:44:03.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:44:03.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:03.788204+0000) 2019-10-18T04:44:03.934 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:44:03.932+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:03.934777+0000) 2019-10-18T04:44:04.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:04.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:04.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:04.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:05.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:05.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:05.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:05.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:06.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:06.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:06.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:06.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:07.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:07.624+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:07.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:07.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:07.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:07.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:07.945 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:44:07.940+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:07.945607+0000) 2019-10-18T04:44:08.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:08.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:08.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:08.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:09.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:09.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:09.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:09.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:10.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:10.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:10.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:10.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:11.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:11.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:11.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:11.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:12.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:12.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:12.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:12.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:12.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:12.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:13.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:44:13.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:13.752442+0000) 2019-10-18T04:44:13.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:13.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:13.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:13.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:13.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:44:13.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:13.788302+0000) 2019-10-18T04:44:13.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:44:13.932+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:13.934990+0000) 2019-10-18T04:44:14.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:14.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:14.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:14.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:15.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:15.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:15.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:15.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:16.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:16.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:16.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:16.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:17.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:17.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:17.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:17.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:17.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:17.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:17.945 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:44:17.940+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:17.945787+0000) 2019-10-18T04:44:18.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:18.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:18.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:18.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:19.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:19.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:19.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:19.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:20.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:20.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:20.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:20.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:21.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:21.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:21.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:21.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:22.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:22.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:22.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:22.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:22.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:22.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:23.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:23.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:23.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:23.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:23.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:44:23.748+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:23.752546+0000) 2019-10-18T04:44:23.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:44:23.784+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:23.788402+0000) 2019-10-18T04:44:23.936 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:44:23.932+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:23.935232+0000) 2019-10-18T04:44:24.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:24.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:24.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:24.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:25.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:25.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:25.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:26.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:26.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:26.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:26.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:27.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:27.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:27.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:27.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:27.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:27.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:27.945 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:44:27.940+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:27.945994+0000) 2019-10-18T04:44:28.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:28.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:28.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:28.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:29.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:29.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:29.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:30.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:30.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:30.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:30.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:31.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:31.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:31.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:31.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:32.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:32.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:32.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:32.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:32.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:32.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:33.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:33.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:33.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:44:33.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:33.752650+0000) 2019-10-18T04:44:33.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:44:33.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:33.788502+0000) 2019-10-18T04:44:33.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:44:33.932+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:33.935463+0000) 2019-10-18T04:44:34.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:34.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:34.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:34.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:35.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:35.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:35.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:35.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:36.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:36.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:36.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:36.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:37.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:37.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:37.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:37.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:37.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:37.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:37.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:44:37.944+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:37.946203+0000) 2019-10-18T04:44:38.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:38.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:38.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:38.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:39.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:39.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:39.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:39.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:40.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:40.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:40.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:40.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:41.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:41.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:41.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:41.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:42.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:42.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:42.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:42.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:42.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:42.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:43.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:43.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:43.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:43.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:43.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:44:43.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:43.752754+0000) 2019-10-18T04:44:43.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:44:43.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:43.788602+0000) 2019-10-18T04:44:43.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:44:43.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:43.935692+0000) 2019-10-18T04:44:44.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:44.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:44.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:44.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:45.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:45.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:45.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:45.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:46.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:46.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:46.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:46.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:47.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:47.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:47.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:47.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:47.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:47.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:47.945 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:44:47.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:47.946362+0000) 2019-10-18T04:44:48.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:48.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:48.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:48.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:49.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:49.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:50.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:50.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:50.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:50.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:51.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:51.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:51.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:51.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:52.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:52.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:52.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:52.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:52.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:52.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:53.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:53.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:53.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:53.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:53.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:44:53.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:53.752855+0000) 2019-10-18T04:44:53.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:44:53.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:53.788702+0000) 2019-10-18T04:44:53.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:44:53.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:53.935907+0000) 2019-10-18T04:44:54.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:54.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:54.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:54.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:55.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:55.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:55.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:55.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:56.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:56.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:56.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:56.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:57.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:44:57.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:44:57.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:57.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:57.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:57.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:57.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:44:57.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:44:57.946540+0000) 2019-10-18T04:44:58.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:58.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:58.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:58.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:44:59.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:59.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:44:59.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:44:59.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:00.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:00.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:00.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:00.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:01.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:01.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:01.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:01.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:02.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:02.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:02.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:02.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:02.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:02.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:03.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:03.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:03.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:03.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:03.752 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:45:03.747+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:03.752960+0000) 2019-10-18T04:45:03.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:45:03.783+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:03.788807+0000) 2019-10-18T04:45:03.935 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:45:03.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:03.936145+0000) 2019-10-18T04:45:04.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:04.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:04.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:04.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:05.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:05.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:05.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:05.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:06.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:06.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:06.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:06.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:07.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:07.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:07.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:07.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:07.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:07.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:07.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:45:07.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:07.946722+0000) 2019-10-18T04:45:08.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:08.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:08.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:08.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:09.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:09.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:09.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:09.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:10.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:10.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:10.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:10.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:11.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:11.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:11.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:11.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:12.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:12.634+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:12.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:12.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:12.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:12.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:13.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:13.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:13.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:13.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:13.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:45:13.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:13.753068+0000) 2019-10-18T04:45:13.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:45:13.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:13.788910+0000) 2019-10-18T04:45:13.937 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:45:13.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:13.936348+0000) 2019-10-18T04:45:14.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:14.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:14.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:14.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:15.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:15.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:15.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:15.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:16.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:16.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:17.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:17.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:17.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:17.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:17.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:17.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:17.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:45:17.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:17.946903+0000) 2019-10-18T04:45:18.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:18.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:18.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:18.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:19.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:19.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:19.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:19.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:20.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:20.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:20.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:20.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:21.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:21.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:21.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:21.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:22.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:22.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:22.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:22.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:22.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:22.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:23.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:45:23.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:23.753169+0000) 2019-10-18T04:45:23.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:23.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:23.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:23.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:23.788 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:45:23.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:23.789008+0000) 2019-10-18T04:45:23.936 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:45:23.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:23.936559+0000) 2019-10-18T04:45:24.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:24.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:24.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:24.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:25.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:25.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:25.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:25.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:26.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:26.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:26.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:26.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:27.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:27.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:27.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:27.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:27.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:27.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:27.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:45:27.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:27.947084+0000) 2019-10-18T04:45:28.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:28.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:28.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:28.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:29.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:29.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:29.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:29.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:30.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:30.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:30.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:30.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:31.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:31.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:31.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:31.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:32.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:32.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:32.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:32.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:32.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:32.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:33.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:45:33.746+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:33.753294+0000) 2019-10-18T04:45:33.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:45:33.782+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:33.789106+0000) 2019-10-18T04:45:33.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:33.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:33.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:33.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:33.936 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:45:33.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:33.936772+0000) 2019-10-18T04:45:34.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:34.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:34.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:34.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:35.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:35.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:35.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:35.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:36.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:36.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:36.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:36.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:37.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:37.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:37.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:37.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:37.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:37.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:37.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:45:37.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:37.947294+0000) 2019-10-18T04:45:38.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:38.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:38.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:38.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:39.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:39.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:39.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:39.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:40.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:40.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:40.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:40.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:41.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:41.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:41.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:41.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:42.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:42.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:42.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:42.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:42.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:42.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:43.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:45:43.745+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:43.753395+0000) 2019-10-18T04:45:43.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:45:43.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:43.789208+0000) 2019-10-18T04:45:43.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:43.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:43.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:43.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:43.936 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:45:43.931+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:43.936985+0000) 2019-10-18T04:45:44.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:44.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:44.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:44.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:45.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:45.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:45.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:45.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:46.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:46.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:46.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:46.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:47.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:47.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:47.946 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:45:47.943+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:47.947476+0000) 2019-10-18T04:45:47.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:47.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:47.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:47.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:48.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:48.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:48.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:48.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:49.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:49.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:49.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:49.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:50.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:50.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:50.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:50.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:51.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:51.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:51.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:51.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:52.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:52.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:52.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:52.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:52.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:52.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:53.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:45:53.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:53.753463+0000) 2019-10-18T04:45:53.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:45:53.781+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:53.789271+0000) 2019-10-18T04:45:53.936 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:45:53.930+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:53.937202+0000) 2019-10-18T04:45:54.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:54.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:54.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:54.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:55.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:55.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:55.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:55.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:56.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:56.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:56.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:56.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:57.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:57.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:57.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:57.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:45:57.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:45:57.947 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:45:57.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:45:57.947690+0000) 2019-10-18T04:45:57.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:57.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:57.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:57.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:45:58.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:58.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:45:58.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:45:58.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:00.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:00.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:00.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:00.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:00.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:00.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:00.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:00.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:02.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:02.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:02.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:02.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:02.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:02.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:03.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:03.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:03.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:03.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:03.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:46:03.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:03.753564+0000) 2019-10-18T04:46:03.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:46:03.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:03.789373+0000) 2019-10-18T04:46:03.936 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:46:03.930+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:03.937364+0000) 2019-10-18T04:46:04.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:04.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:04.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:04.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:05.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:05.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:05.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:05.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:06.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:06.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:06.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:06.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:07.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:07.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:07.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:07.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:07.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:07.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:07.947 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:46:07.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:07.947901+0000) 2019-10-18T04:46:08.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:08.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:08.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:08.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:09.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:09.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:09.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:09.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:10.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:10.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:10.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:10.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:11.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:11.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:11.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:11.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:12.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:12.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:12.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:12.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:12.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:12.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:13.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:13.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:13.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:13.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:13.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:46:13.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:13.753670+0000) 2019-10-18T04:46:13.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:46:13.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:13.789470+0000) 2019-10-18T04:46:13.937 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:46:13.930+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:13.937575+0000) 2019-10-18T04:46:14.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:14.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:14.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:14.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:15.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:15.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:15.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:15.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:16.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:16.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:16.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:16.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:17.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:17.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:17.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:17.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:17.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:17.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:17.947 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:46:17.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:17.948112+0000) 2019-10-18T04:46:18.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:18.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:18.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:18.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:19.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:19.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:19.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:19.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:20.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:20.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:20.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:20.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:20.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:20.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:20.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:20.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:22.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:22.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:22.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:22.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:22.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:22.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:23.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:23.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:23.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:23.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:23.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:46:23.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:23.753780+0000) 2019-10-18T04:46:23.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:46:23.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:23.789572+0000) 2019-10-18T04:46:23.937 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:46:23.930+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:23.937792+0000) 2019-10-18T04:46:23.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:23.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:23.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:23.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:24.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:24.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:35:53.003168+0000 (oldest deadline 2019-10-18T04:36:13.003168+0000) 2019-10-18T04:46:24.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:24.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:26.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:26.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:27.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:27.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:27.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:27.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:27.948 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:46:27.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:27.948296+0000) 2019-10-18T04:46:28.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:28.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:29.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:29.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:30.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:30.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:31.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:31.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:32.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:32.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:32.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:32.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:33.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:33.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:46:33.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:33.753881+0000) 2019-10-18T04:46:33.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:46:33.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:33.789671+0000) 2019-10-18T04:46:33.937 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:46:33.930+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:33.937995+0000) 2019-10-18T04:46:34.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:34.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:34.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:34.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:35.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:35.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:36.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:36.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:37.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:37.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:37.947 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:46:37.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:37.948476+0000) 2019-10-18T04:46:37.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:37.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:38.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:38.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:39.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:39.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:40.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:40.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:41.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:41.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:42.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:42.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:42.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:42.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:43.753 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:46:43.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:43.753981+0000) 2019-10-18T04:46:43.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:46:43.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:43.789769+0000) 2019-10-18T04:46:43.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:43.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:43.937 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:46:43.934+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:43.938292+0000) 2019-10-18T04:46:44.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:44.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:45.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:45.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:46.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:46.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:47.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:47.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:47.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:47.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:47.948 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:46:47.942+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:47.948662+0000) 2019-10-18T04:46:48.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:48.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:48.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:48.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:49.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:49.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:49.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:49.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:50.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:50.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:50.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:50.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:51.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:51.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:51.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:51.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:52.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:52.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:52.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:52.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:52.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:52.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:53.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:46:53.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:53.754077+0000) 2019-10-18T04:46:53.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:46:53.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:53.789866+0000) 2019-10-18T04:46:53.937 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:46:53.938+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:53.938497+0000) 2019-10-18T04:46:53.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:53.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:53.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:53.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:54.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:54.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:55.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:54.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:56.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:56.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:56.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:56.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:57.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:57.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:57.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:57.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:57.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:46:57.655+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:46:57.948 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:46:57.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:46:57.948841+0000) 2019-10-18T04:46:58.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:58.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:58.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:58.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:59.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:58.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:59.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:58.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:46:59.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:59.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:46:59.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:46:59.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:01.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:00.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:01.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:00.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:01.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:01.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:01.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:01.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:02.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:02.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:03.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:03.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:03.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:03.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:03.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:47:03.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:03.754182+0000) 2019-10-18T04:47:03.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:47:03.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:03.789967+0000) 2019-10-18T04:47:03.938 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:03.937+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:03.938693+0000) 2019-10-18T04:47:03.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:03.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:03.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:03.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:05.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:04.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:05.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:04.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:06.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:06.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:06.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:06.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:07.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:07.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:07.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:07.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:07.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:07.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:07.948 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:47:07.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:07.949025+0000) 2019-10-18T04:47:07.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:07.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:07.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:07.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:08.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:08.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:08.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:08.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:09.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:09.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:09.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:09.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:10.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:10.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:10.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:10.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:12.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:12.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:12.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:12.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:12.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:12.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:13.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:13.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:13.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:13.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:13.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:47:13.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:13.754282+0000) 2019-10-18T04:47:13.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:47:13.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:13.790063+0000) 2019-10-18T04:47:13.938 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:13.937+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:13.938921+0000) 2019-10-18T04:47:13.939 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:13.937+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:13.940082+0000) 2019-10-18T04:47:14.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:14.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:14.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:14.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:15.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:15.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:15.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:15.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:16.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:16.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:16.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:16.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:17.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:17.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:17.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:17.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:17.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:17.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:17.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:47:17.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:17.949204+0000) 2019-10-18T04:47:18.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:18.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:18.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:18.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:18.438 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:18.437+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:18.439315+0000) 2019-10-18T04:47:19.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:19.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:19.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:19.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:20.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:20.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:20.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:20.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:21.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:21.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:21.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:21.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:22.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:22.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:22.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:22.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:22.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:22.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:23.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:23.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:23.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:23.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:23.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:47:23.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:23.754385+0000) 2019-10-18T04:47:23.789 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:47:23.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:23.790160+0000) 2019-10-18T04:47:24.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:24.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:24.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:24.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:25.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:25.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:25.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:25.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:25.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:25.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:25.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:25.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:26.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:26.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:26.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:26.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:27.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:27.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:27.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:27.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:27.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:27.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:27.948 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:47:27.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:27.949396+0000) 2019-10-18T04:47:28.438 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:28.437+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:28.439496+0000) 2019-10-18T04:47:28.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:28.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:28.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:28.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:29.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:29.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:29.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:29.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:30.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:30.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:30.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:30.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:31.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:31.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:31.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:31.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:32.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:32.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:32.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:32.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:33.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:47:33.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:33.754485+0000) 2019-10-18T04:47:33.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:47:33.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:33.790256+0000) 2019-10-18T04:47:33.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:33.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:33.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:33.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:35.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:35.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:35.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:35.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:35.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:35.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:35.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:35.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:36.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:36.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:36.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:36.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:37.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:37.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:37.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:37.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:37.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:37.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:37.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:47:37.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:37.949584+0000) 2019-10-18T04:47:38.439 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:38.437+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:38.439676+0000) 2019-10-18T04:47:38.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:38.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:38.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:38.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:39.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:39.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:39.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:39.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:40.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:40.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:40.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:40.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:41.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:41.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:41.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:41.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:42.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:42.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:42.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:42.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:42.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:42.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:43.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:47:43.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:43.754594+0000) 2019-10-18T04:47:43.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:47:43.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:43.790357+0000) 2019-10-18T04:47:43.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:43.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:43.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:43.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:44.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:44.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:45.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:45.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:45.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:45.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:46.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:46.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:46.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:46.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:47.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:47.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:47.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:47.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:47.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:47.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:47.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:47:47.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:47.949763+0000) 2019-10-18T04:47:48.439 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:48.437+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:48.439856+0000) 2019-10-18T04:47:48.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:48.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:48.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:48.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:49.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:49.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:49.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:49.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:50.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:50.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:50.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:50.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:51.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:51.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:51.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:51.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:52.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:52.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:52.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:52.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:52.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:52.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:53.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:53.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:53.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:53.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:53.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:47:53.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:53.754701+0000) 2019-10-18T04:47:53.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:47:53.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:53.790456+0000) 2019-10-18T04:47:54.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:54.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:54.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:54.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:55.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:55.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:55.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:55.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:56.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:56.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:56.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:56.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:57.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:47:57.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:47:57.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:57.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:57.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:57.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:57.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:47:57.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:57.949946+0000) 2019-10-18T04:47:58.439 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:47:58.437+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:47:58.440039+0000) 2019-10-18T04:47:58.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:58.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:58.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:58.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:47:59.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:59.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:47:59.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:47:59.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:00.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:00.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:00.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:00.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:01.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:01.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:01.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:01.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:02.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:02.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:02.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:02.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:02.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:02.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:03.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:03.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:03.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:03.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:03.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:48:03.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:03.754804+0000) 2019-10-18T04:48:03.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:48:03.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:03.790555+0000) 2019-10-18T04:48:04.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:04.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:04.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:04.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:05.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:05.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:05.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:05.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:06.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:06.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:06.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:06.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:07.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:07.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:07.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:07.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:07.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:07.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:07.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:48:07.945+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:07.950127+0000) 2019-10-18T04:48:08.440 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:48:08.437+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:08.440217+0000) 2019-10-18T04:48:08.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:08.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:08.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:08.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:09.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:09.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:09.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:09.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:10.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:10.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:10.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:10.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:11.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:11.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:11.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:11.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:12.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:12.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:12.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:12.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:12.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:12.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:13.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:13.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:13.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:13.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:13.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:48:13.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:13.754905+0000) 2019-10-18T04:48:13.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:48:13.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:13.790651+0000) 2019-10-18T04:48:14.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:14.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:14.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:14.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:15.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:15.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:15.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:15.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:16.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:16.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:16.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:16.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:17.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:17.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:17.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:17.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:17.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:17.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:17.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:48:17.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:17.950336+0000) 2019-10-18T04:48:18.439 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:48:18.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:18.440398+0000) 2019-10-18T04:48:18.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:18.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:18.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:18.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:19.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:19.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:19.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:19.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:20.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:20.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:20.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:20.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:21.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:21.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:21.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:21.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:22.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:22.673+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:22.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:22.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:22.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:22.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:23.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:23.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:23.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:23.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:23.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:48:23.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:23.755009+0000) 2019-10-18T04:48:23.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:48:23.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:23.790752+0000) 2019-10-18T04:48:24.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:24.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:24.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:24.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:25.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:25.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:25.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:25.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:26.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:26.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:26.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:26.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:27.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:27.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:27.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:27.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:27.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:27.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:27.949 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:48:27.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:27.950520+0000) 2019-10-18T04:48:28.440 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:48:28.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:28.440575+0000) 2019-10-18T04:48:28.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:28.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:28.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:28.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:29.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:29.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:29.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:29.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:30.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:30.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:30.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:30.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:31.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:31.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:31.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:31.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:32.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:32.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:32.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:32.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:32.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:32.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:33.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:33.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:33.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:33.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:33.754 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:48:33.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:33.755113+0000) 2019-10-18T04:48:33.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:48:33.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:33.790856+0000) 2019-10-18T04:48:34.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:34.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:34.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:34.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:35.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:35.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:35.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:35.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:36.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:36.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:36.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:36.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:37.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:37.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:37.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:37.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:37.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:37.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:37.950 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:48:37.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:37.950706+0000) 2019-10-18T04:48:38.440 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:48:38.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:38.440763+0000) 2019-10-18T04:48:38.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:38.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:38.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:38.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:39.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:39.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:39.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:39.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:40.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:40.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:40.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:40.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:41.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:41.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:41.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:41.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:42.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:42.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:42.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:42.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:42.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:42.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:43.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:43.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:43.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:43.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:43.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:48:43.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:43.755219+0000) 2019-10-18T04:48:43.790 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:48:43.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:43.790956+0000) 2019-10-18T04:48:44.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:44.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:44.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:44.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:45.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:45.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:45.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:45.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:46.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:46.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:47.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:47.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:47.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:47.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:47.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:47.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:47.950 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:48:47.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:47.950882+0000) 2019-10-18T04:48:48.440 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:48:48.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:48.440951+0000) 2019-10-18T04:48:48.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:48.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:48.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:48.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:49.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:49.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:49.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:49.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:50.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:50.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:50.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:50.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:51.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:51.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:51.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:51.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:52.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:52.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:52.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:52.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:52.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:52.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:53.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:53.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:53.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:53.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:53.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:48:53.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:53.755321+0000) 2019-10-18T04:48:53.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:48:53.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:53.791055+0000) 2019-10-18T04:48:54.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:54.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:54.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:54.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:55.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:55.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:55.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:55.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:56.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:56.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:56.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:56.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:57.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:57.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:57.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:57.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:57.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:48:57.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:48:57.950 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:48:57.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:57.951074+0000) 2019-10-18T04:48:58.440 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:48:58.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:48:58.441130+0000) 2019-10-18T04:48:58.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:58.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:58.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:58.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:48:59.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:59.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:48:59.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:48:59.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:00.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:00.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:00.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:00.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:01.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:01.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:01.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:01.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:02.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:02.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:02.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:02.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:02.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:02.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:03.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:03.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:03.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:03.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:03.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:49:03.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:03.755429+0000) 2019-10-18T04:49:03.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:49:03.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:03.791159+0000) 2019-10-18T04:49:04.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:04.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:04.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:04.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:05.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:05.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:05.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:05.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:06.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:06.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:06.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:06.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:07.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:07.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:07.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:07.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:07.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:07.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:07.950 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:49:07.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:07.951261+0000) 2019-10-18T04:49:08.440 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:49:08.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:08.441314+0000) 2019-10-18T04:49:08.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:08.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:08.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:08.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:09.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:09.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:09.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:10.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:10.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:10.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:10.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:11.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:11.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:11.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:11.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:12.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:12.679+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:12.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:12.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:12.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:12.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:13.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:13.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:13.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:13.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:13.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:49:13.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:13.755538+0000) 2019-10-18T04:49:13.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:49:13.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:13.791260+0000) 2019-10-18T04:49:14.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:14.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:14.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:14.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:15.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:15.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:15.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:15.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:16.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:16.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:16.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:16.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:17.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:17.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:17.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:17.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:17.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:17.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:17.950 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:49:17.948+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:17.951443+0000) 2019-10-18T04:49:18.441 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:49:18.436+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:18.441499+0000) 2019-10-18T04:49:18.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:18.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:18.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:18.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:19.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:19.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:19.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:19.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:20.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:20.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:20.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:20.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:21.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:21.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:21.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:21.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:22.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:22.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:22.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:22.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:22.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:23.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:49:23.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:23.755638+0000) 2019-10-18T04:49:23.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:23.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:23.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:23.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:23.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:49:23.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:23.791360+0000) 2019-10-18T04:49:24.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:24.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:24.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:24.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:25.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:25.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:25.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:25.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:26.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:26.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:26.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:26.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:27.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:27.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:27.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:27.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:27.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:27.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:27.951 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:49:27.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:27.951662+0000) 2019-10-18T04:49:28.441 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:49:28.435+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:28.441687+0000) 2019-10-18T04:49:28.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:28.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:28.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:28.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:29.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:29.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:29.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:29.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:30.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:30.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:30.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:30.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:31.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:31.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:31.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:31.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:32.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:32.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:32.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:32.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:32.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:32.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:33.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:49:33.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:33.755745+0000) 2019-10-18T04:49:33.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:49:33.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:33.791462+0000) 2019-10-18T04:49:33.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:33.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:33.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:33.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:34.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:34.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:34.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:34.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:35.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:35.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:35.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:35.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:36.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:36.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:36.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:36.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:37.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:37.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:37.951 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:49:37.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:37.951876+0000) 2019-10-18T04:49:37.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:37.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:37.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:37.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:38.441 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:49:38.435+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:38.441873+0000) 2019-10-18T04:49:39.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:39.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:39.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:39.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:39.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:39.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:39.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:39.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:40.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:40.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:40.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:40.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:41.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:41.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:41.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:41.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:42.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:42.687+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:42.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:42.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:42.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:42.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:43.755 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:49:43.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:43.755852+0000) 2019-10-18T04:49:43.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:49:43.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:43.791566+0000) 2019-10-18T04:49:43.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:43.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:43.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:43.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:44.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:44.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:44.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:44.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:45.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:45.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:45.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:45.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:47.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:47.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:47.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:47.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:47.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:47.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:47.951 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:49:47.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:47.952087+0000) 2019-10-18T04:49:47.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:47.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:47.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:47.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:48.441 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:49:48.435+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:48.442056+0000) 2019-10-18T04:49:48.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:48.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:48.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:48.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:49.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:49.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:49.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:49.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:51.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:51.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:52.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:52.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:52.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:52.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:52.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:52.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:53.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:53.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:53.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:53.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:53.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:49:53.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:53.755955+0000) 2019-10-18T04:49:53.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:49:53.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:53.791669+0000) 2019-10-18T04:49:53.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:53.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:53.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:53.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:54.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:54.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:54.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:54.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:55.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:55.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:55.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:55.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:57.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:56.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:57.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:56.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:57.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:49:57.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:49:57.951 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:49:57.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:57.952274+0000) 2019-10-18T04:49:58.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:58.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:58.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:58.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:49:58.441 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:49:58.439+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:49:58.442235+0000) 2019-10-18T04:49:58.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:58.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:49:58.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:49:58.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:00.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:00.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:00.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:00.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:01.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:01.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:01.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:01.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:02.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:02.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:02.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:02.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:03.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:03.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:03.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:03.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:03.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:50:03.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:03.756059+0000) 2019-10-18T04:50:03.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:50:03.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:03.791776+0000) 2019-10-18T04:50:04.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:04.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:04.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:04.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:05.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:05.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:05.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:05.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:06.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:06.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:06.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:06.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:07.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:07.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:07.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:07.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:07.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:07.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:07.951 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:50:07.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:07.952460+0000) 2019-10-18T04:50:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:08.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:08.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:08.441 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:50:08.439+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:08.442411+0000) 2019-10-18T04:50:09.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:09.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:09.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:09.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:10.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:10.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:10.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:10.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:11.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:11.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:11.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:11.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:12.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:12.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:12.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:12.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:12.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:12.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:13.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:13.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:13.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:13.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:13.772 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:50:13.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:13.756176+0000) 2019-10-18T04:50:13.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:50:13.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:13.791878+0000) 2019-10-18T04:50:14.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:14.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:14.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:14.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:15.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:15.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:15.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:15.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:16.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:16.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:16.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:16.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:17.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:17.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:17.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:17.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:17.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:17.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:17.952 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:50:17.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:17.952639+0000) 2019-10-18T04:50:18.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:18.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:18.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:18.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:18.442 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:50:18.439+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:18.442600+0000) 2019-10-18T04:50:19.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:19.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:19.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:19.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:20.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:20.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:20.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:20.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:21.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:21.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:21.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:21.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:22.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:22.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:22.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:22.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:22.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:22.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:23.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:23.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:23.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:50:23.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:23.756279+0000) 2019-10-18T04:50:23.791 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:50:23.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:23.791986+0000) 2019-10-18T04:50:24.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:24.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:24.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:24.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:24.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:24.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:24.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:24.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:25.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:25.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:25.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:25.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:26.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:26.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:26.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:26.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:27.703 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:27.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:27.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:27.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:27.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:27.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:27.953 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:50:27.947+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:27.952816+0000) 2019-10-18T04:50:28.442 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:50:28.439+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:28.442786+0000) 2019-10-18T04:50:28.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:28.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:28.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:28.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:29.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:29.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:29.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:29.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:30.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:30.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:30.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:30.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:31.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:31.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:31.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:31.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:32.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:32.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:32.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:32.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:32.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:32.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:33.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:50:33.749+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:33.756378+0000) 2019-10-18T04:50:33.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:50:33.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:33.792086+0000) 2019-10-18T04:50:33.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:33.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:33.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:33.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:34.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:34.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:34.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:34.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:35.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:35.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:35.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:35.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:36.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:36.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:36.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:36.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:37.705 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:37.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:37.952 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:50:37.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:37.953004+0000) 2019-10-18T04:50:37.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:37.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:37.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:37.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:38.442 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:50:38.438+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:38.442963+0000) 2019-10-18T04:50:38.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:38.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:38.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:38.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:40.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:40.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:40.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:40.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:40.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:40.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:40.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:40.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:41.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:41.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:41.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:41.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:42.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:42.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:42.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:42.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:42.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:42.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:43.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:50:43.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:43.756481+0000) 2019-10-18T04:50:43.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:50:43.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:43.792187+0000) 2019-10-18T04:50:43.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:43.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:43.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:43.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:44.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:44.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:44.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:44.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:45.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:45.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:45.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:45.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:46.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:46.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:46.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:46.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:47.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:47.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:47.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:47.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:47.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:47.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:47.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:50:47.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:47.953219+0000) 2019-10-18T04:50:48.442 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:50:48.438+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:48.443148+0000) 2019-10-18T04:50:48.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:48.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:48.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:48.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:49.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:49.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:49.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:49.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:50.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:50.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:50.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:50.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:51.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:51.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:51.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:51.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:52.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:52.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:52.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:52.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:52.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:52.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:53.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:53.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:53.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:53.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:53.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:50:53.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:53.756582+0000) 2019-10-18T04:50:53.792 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:50:53.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:53.792284+0000) 2019-10-18T04:50:54.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:54.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:54.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:54.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:55.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:55.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:55.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:55.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:56.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:56.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:56.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:56.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:57.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:50:57.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:50:57.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:57.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:57.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:57.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:57.952 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:50:57.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:57.953400+0000) 2019-10-18T04:50:58.442 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:50:58.438+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:50:58.443329+0000) 2019-10-18T04:50:58.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:58.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:58.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:58.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:50:59.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:59.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:50:59.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:50:59.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:00.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:00.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:00.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:00.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:01.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:01.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:01.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:01.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:02.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:02.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:02.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:02.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:02.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:02.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:03.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:03.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:03.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:03.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:03.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:51:03.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:03.756681+0000) 2019-10-18T04:51:03.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:51:03.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:03.792389+0000) 2019-10-18T04:51:04.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:04.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:04.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:04.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:05.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:05.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:05.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:05.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:06.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:06.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:06.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:06.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:07.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:07.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:07.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:07.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:07.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:07.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:07.953 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:51:07.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:07.953611+0000) 2019-10-18T04:51:08.442 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:51:08.438+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:08.443509+0000) 2019-10-18T04:51:08.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:08.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:08.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:08.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:09.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:09.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:09.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:09.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:10.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:10.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:10.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:10.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:11.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:11.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:11.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:11.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:12.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:12.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:12.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:12.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:12.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:12.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:13.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:13.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:13.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:13.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:13.756 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:51:13.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:13.756783+0000) 2019-10-18T04:51:13.792 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:51:13.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:13.792491+0000) 2019-10-18T04:51:14.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:14.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:14.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:14.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:15.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:15.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:15.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:15.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:16.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:16.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:16.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:16.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:17.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:17.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:17.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:17.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:17.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:17.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:17.955 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:51:17.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:17.953793+0000) 2019-10-18T04:51:18.443 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:51:18.438+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:18.443694+0000) 2019-10-18T04:51:18.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:18.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:18.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:18.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:19.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:19.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:19.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:19.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:20.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:20.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:20.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:20.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:21.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:21.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:21.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:21.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:22.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:22.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:22.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:22.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:22.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:22.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:23.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:23.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:23.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:23.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:23.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:51:23.752+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:23.756884+0000) 2019-10-18T04:51:23.792 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:51:23.788+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:23.792594+0000) 2019-10-18T04:51:24.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:24.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:24.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:24.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:25.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:25.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:25.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:25.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:26.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:26.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:26.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:27.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:27.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:27.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:27.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:27.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:27.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:27.953 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:51:27.946+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:27.953974+0000) 2019-10-18T04:51:28.443 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:51:28.438+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:28.443873+0000) 2019-10-18T04:51:28.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:28.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:28.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:28.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:29.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:29.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:29.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:29.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:30.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:30.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:30.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:30.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:31.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:31.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:31.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:31.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:32.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:32.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:32.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:32.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:32.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:32.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:33.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:33.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:33.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:33.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:33.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:51:33.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:33.756980+0000) 2019-10-18T04:51:33.792 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:51:33.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:33.792693+0000) 2019-10-18T04:51:34.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:34.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:34.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:34.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:35.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:35.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:35.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:35.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:36.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:36.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:36.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:36.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:37.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:37.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:37.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:37.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:37.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:37.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:37.953 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:51:37.950+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:37.954159+0000) 2019-10-18T04:51:38.443 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:51:38.442+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:38.444058+0000) 2019-10-18T04:51:38.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:38.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:38.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:38.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:39.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:39.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:39.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:39.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:40.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:40.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:40.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:40.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:41.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:41.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:41.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:41.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:42.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:42.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:42.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:42.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:42.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:43.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:43.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:43.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:43.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:43.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:51:43.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:43.757078+0000) 2019-10-18T04:51:43.792 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:51:43.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:43.792795+0000) 2019-10-18T04:51:44.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:44.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:44.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:44.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:45.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:45.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:45.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:45.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:46.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:46.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:46.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:46.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:47.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:47.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:47.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:47.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:47.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:47.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:47.953 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:51:47.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:47.954349+0000) 2019-10-18T04:51:48.443 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:51:48.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:48.444256+0000) 2019-10-18T04:51:48.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:48.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:48.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:48.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:49.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:49.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:49.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:49.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:50.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:50.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:50.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:50.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:51.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:51.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:51.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:51.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:52.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:52.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:52.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:52.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:52.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:52.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:53.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:53.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:53.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:53.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:53.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:51:53.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:53.757183+0000) 2019-10-18T04:51:53.792 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:51:53.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:53.792910+0000) 2019-10-18T04:51:54.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:54.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:54.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:54.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:55.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:55.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:55.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:55.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:56.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:56.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:56.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:56.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:57.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:57.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:57.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:57.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:57.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:51:57.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:51:57.954 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:51:57.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:57.954541+0000) 2019-10-18T04:51:58.443 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:51:58.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:51:58.444438+0000) 2019-10-18T04:51:58.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:58.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:58.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:58.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:51:59.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:59.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:51:59.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:51:59.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:00.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:00.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:00.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:00.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:01.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:01.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:01.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:01.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:02.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:02.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:02.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:02.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:02.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:02.715+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:03.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:03.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:03.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:03.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:03.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:52:03.751+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:03.757282+0000) 2019-10-18T04:52:03.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:52:03.787+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:03.793013+0000) 2019-10-18T04:52:04.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:04.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:04.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:04.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:05.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:05.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:05.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:06.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:06.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:06.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:06.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:07.719 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:07.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:07.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:07.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:07.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:07.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:07.954 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:52:07.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:07.954728+0000) 2019-10-18T04:52:08.444 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:52:08.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:08.444622+0000) 2019-10-18T04:52:08.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:08.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:08.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:08.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:09.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:09.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:09.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:09.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:10.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:10.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:10.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:10.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:11.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:11.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:11.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:11.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:12.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:12.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:12.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:12.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:12.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:12.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:13.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:13.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:13.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:13.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:13.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:52:13.750+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:13.757383+0000) 2019-10-18T04:52:13.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:52:13.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:13.793115+0000) 2019-10-18T04:52:14.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:14.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:14.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:14.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:15.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:15.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:15.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:15.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:16.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:16.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:16.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:16.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:17.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:17.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:17.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:17.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:17.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:17.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:17.954 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:52:17.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:17.954906+0000) 2019-10-18T04:52:18.444 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:52:18.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:18.444803+0000) 2019-10-18T04:52:18.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:18.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:18.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:18.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:19.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:19.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:19.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:19.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:20.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:20.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:20.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:20.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:21.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:21.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:21.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:21.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:22.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:22.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:22.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:22.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:22.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:22.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:23.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:52:23.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:23.757464+0000) 2019-10-18T04:52:23.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:23.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:23.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:23.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:23.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:52:23.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:23.793219+0000) 2019-10-18T04:52:24.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:24.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:24.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:24.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:25.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:25.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:25.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:25.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:26.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:26.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:26.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:26.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:27.724 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:27.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:27.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:27.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:27.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:27.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:27.954 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:52:27.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:27.955093+0000) 2019-10-18T04:52:28.444 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:52:28.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:28.444981+0000) 2019-10-18T04:52:28.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:28.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:28.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:28.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:29.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:29.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:29.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:29.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:30.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:30.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:30.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:30.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:31.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:31.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:31.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:31.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:32.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:32.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:32.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:32.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:32.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:32.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:33.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:33.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:33.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:33.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:33.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:52:33.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:33.757563+0000) 2019-10-18T04:52:33.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:52:33.786+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:33.793323+0000) 2019-10-18T04:52:34.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:34.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:34.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:34.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:35.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:35.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:35.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:35.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:36.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:36.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:36.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:36.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:37.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:37.722+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:37.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:37.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:37.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:37.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:37.955 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:52:37.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:37.955278+0000) 2019-10-18T04:52:38.445 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:52:38.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:38.445158+0000) 2019-10-18T04:52:38.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:38.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:38.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:38.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:39.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:39.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:39.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:39.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:40.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:40.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:40.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:40.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:41.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:41.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:41.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:41.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:42.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:42.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:42.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:42.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:42.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:42.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:43.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:52:43.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:43.757665+0000) 2019-10-18T04:52:43.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:43.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:43.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:43.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:43.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:52:43.785+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:43.793428+0000) 2019-10-18T04:52:44.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:44.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:44.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:44.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:45.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:45.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:45.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:45.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:46.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:46.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:46.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:46.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:47.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:47.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:47.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:47.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:47.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:47.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:47.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:52:47.953+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:47.955491+0000) 2019-10-18T04:52:48.444 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:52:48.441+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:48.445343+0000) 2019-10-18T04:52:48.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:48.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:48.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:48.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:49.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:49.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:49.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:49.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:50.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:50.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:50.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:50.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:51.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:51.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:51.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:51.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:52.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:52.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:52.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:52.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:52.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:52.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:53.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:53.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:53.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:53.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:53.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:52:53.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:53.757765+0000) 2019-10-18T04:52:53.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:52:53.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:53.793539+0000) 2019-10-18T04:52:54.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:54.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:54.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:54.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:55.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:55.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:55.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:55.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:56.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:56.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:56.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:56.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:57.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:52:57.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:52:57.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:57.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:57.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:57.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:57.957 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:52:57.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:57.955674+0000) 2019-10-18T04:52:58.445 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:52:58.440+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:52:58.445525+0000) 2019-10-18T04:52:58.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:58.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:58.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:58.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:52:59.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:59.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:52:59.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:52:59.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:00.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:00.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:00.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:01.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:01.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:01.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:01.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:02.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:02.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:02.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:02.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:02.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:02.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:03.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:03.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:03.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:03.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:03.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:53:03.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:03.757860+0000) 2019-10-18T04:53:03.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:53:03.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:03.793638+0000) 2019-10-18T04:53:04.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:04.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:04.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:04.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:05.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:05.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:05.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:05.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:06.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:06.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:06.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:06.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:07.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:07.729+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:07.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:07.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:07.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:07.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:07.955 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:53:07.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:07.955863+0000) 2019-10-18T04:53:08.449 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:08.440+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:08.445665+0000) 2019-10-18T04:53:08.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:08.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:08.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:08.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:09.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:09.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:09.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:09.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:10.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:10.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:10.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:10.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:11.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:11.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:11.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:11.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:12.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:12.733+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:12.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:12.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:12.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:12.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:13.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:13.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:13.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:13.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:13.757 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:53:13.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:13.757958+0000) 2019-10-18T04:53:13.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:53:13.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:13.793740+0000) 2019-10-18T04:53:14.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:14.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:14.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:14.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:15.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:15.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:15.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:15.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:16.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:16.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:16.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:16.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:17.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:17.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:17.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:17.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:17.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:17.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:17.955 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:53:17.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:17.956039+0000) 2019-10-18T04:53:18.445 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:18.440+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:18.445847+0000) 2019-10-18T04:53:18.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:18.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:18.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:18.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:19.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:19.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:19.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:19.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:20.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:20.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:20.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:20.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:21.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:21.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:21.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:21.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:22.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:22.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:22.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:22.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:22.737 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:22.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:23.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:23.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:23.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:23.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:23.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:53:23.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:23.758061+0000) 2019-10-18T04:53:23.793 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:53:23.792+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:23.793841+0000) 2019-10-18T04:53:24.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:24.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:24.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:24.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:25.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:25.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:25.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:25.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:26.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:26.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:26.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:26.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:27.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:27.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:27.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:27.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:27.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:27.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:27.955 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:53:27.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:27.956221+0000) 2019-10-18T04:53:28.445 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:28.440+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:28.446046+0000) 2019-10-18T04:53:28.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:28.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:28.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:28.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:29.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:29.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:29.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:29.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:30.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:30.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:30.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:30.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:31.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:31.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:31.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:31.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:32.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:32.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:32.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:32.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:32.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:32.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:33.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:53:33.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:33.758161+0000) 2019-10-18T04:53:33.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:33.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:33.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:33.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:33.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:53:33.792+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:33.793939+0000) 2019-10-18T04:53:34.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:34.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:34.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:34.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:35.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:35.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:35.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:35.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:36.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:36.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:36.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:36.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:37.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:37.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:37.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:37.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:37.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:37.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:37.955 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:53:37.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:37.956401+0000) 2019-10-18T04:53:38.445 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:38.444+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:38.446261+0000) 2019-10-18T04:53:38.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:38.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:38.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:38.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:39.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:39.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:39.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:39.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:40.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:40.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:40.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:40.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:41.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:41.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:41.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:41.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:42.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:42.736+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:42.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:42.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:42.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:42.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:43.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:53:43.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:43.758256+0000) 2019-10-18T04:53:43.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:53:43.792+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:43.794035+0000) 2019-10-18T04:53:43.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:43.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:43.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:43.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:44.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:44.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:45.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:45.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:45.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:45.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:46.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:46.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:46.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:46.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:47.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:47.740+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:47.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:47.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:47.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:47.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:47.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:53:47.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:47.956614+0000) 2019-10-18T04:53:48.446 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:48.444+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:48.446430+0000) 2019-10-18T04:53:48.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:48.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:48.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:48.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:49.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:49.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:49.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:49.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:50.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:50.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:50.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:50.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:51.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:51.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:51.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:51.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:52.743 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:52.739+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:52.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:52.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:52.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:52.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:53.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:53:53.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:53.758355+0000) 2019-10-18T04:53:53.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:53:53.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:53.794136+0000) 2019-10-18T04:53:53.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:53.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:53.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:53.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:54.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:54.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:54.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:54.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:55.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:55.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:55.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:55.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:56.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:56.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:56.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:56.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:57.743 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:53:57.739+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:53:57.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:57.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:57.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:57.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:57.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:53:57.952+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:57.956797+0000) 2019-10-18T04:53:58.446 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:58.444+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:58.446597+0000) 2019-10-18T04:53:58.446 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:53:58.444+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:53:58.447298+0000) 2019-10-18T04:53:58.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:58.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:58.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:58.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:53:59.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:59.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:53:59.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:53:59.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:00.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:00.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:00.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:00.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:01.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:01.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:01.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:01.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:02.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:02.739+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:02.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:02.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:02.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:02.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:02.947 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:54:02.944+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:02.947013+0000) 2019-10-18T04:54:03.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:54:03.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:03.758452+0000) 2019-10-18T04:54:03.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:54:03.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:03.794441+0000) 2019-10-18T04:54:03.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:03.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:03.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:03.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:04.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:04.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:04.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:04.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:05.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:05.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:05.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:05.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:06.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:06.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:06.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:06.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:07.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:07.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:07.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:07.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:07.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:07.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:07.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:54:07.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:07.956986+0000) 2019-10-18T04:54:08.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:08.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:08.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:08.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:09.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:09.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:09.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:09.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:10.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:10.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:10.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:10.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:11.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:11.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:11.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:11.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:12.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:12.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:12.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:12.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:12.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:12.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:12.946 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:54:12.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:12.947204+0000) 2019-10-18T04:54:13.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:13.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:13.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:13.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:13.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:54:13.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:13.758550+0000) 2019-10-18T04:54:13.794 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:54:13.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:13.794539+0000) 2019-10-18T04:54:14.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:14.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:14.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:14.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:15.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:15.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:15.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:15.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:16.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:16.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:16.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:16.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:17.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:17.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:17.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:17.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:17.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:17.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:17.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:54:17.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:17.957196+0000) 2019-10-18T04:54:18.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:18.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:18.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:18.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:19.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:19.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:19.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:19.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:20.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:20.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:20.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:20.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:21.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:21.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:21.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:21.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:22.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:22.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:22.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:22.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:22.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:22.743+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:22.946 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:54:22.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:22.947391+0000) 2019-10-18T04:54:23.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:23.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:23.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:23.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:23.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:54:23.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:23.758653+0000) 2019-10-18T04:54:23.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:54:23.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:23.795000+0000) 2019-10-18T04:54:24.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:24.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:24.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:24.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:25.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:25.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:25.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:25.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:26.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:26.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:26.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:26.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:27.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:27.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:27.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:27.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:27.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:27.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:27.956 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:54:27.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:27.957375+0000) 2019-10-18T04:54:28.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:28.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:28.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:28.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:29.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:29.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:30.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:30.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:30.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:30.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:31.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:31.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:31.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:31.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:32.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:32.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:32.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:32.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:32.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:32.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:32.947 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:54:32.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:32.947589+0000) 2019-10-18T04:54:33.758 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:54:33.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:33.758758+0000) 2019-10-18T04:54:33.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:54:33.790+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:33.795101+0000) 2019-10-18T04:54:33.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:33.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:33.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:33.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:34.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:34.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:34.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:34.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:35.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:35.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:35.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:35.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:36.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:36.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:36.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:36.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:37.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:37.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:37.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:37.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:37.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:37.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:37.957 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:54:37.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:37.957568+0000) 2019-10-18T04:54:38.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:38.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:38.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:38.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:39.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:39.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:39.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:39.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:40.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:40.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:40.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:40.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:41.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:41.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:41.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:41.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:42.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:42.746+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:42.947 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:54:42.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:42.947791+0000) 2019-10-18T04:54:42.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:42.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:42.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:42.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:43.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:54:43.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:43.758861+0000) 2019-10-18T04:54:43.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:54:43.790+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:43.795197+0000) 2019-10-18T04:54:43.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:43.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:43.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:43.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:44.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:44.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:44.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:44.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:45.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:45.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:45.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:45.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:46.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:46.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:46.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:46.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:47.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:47.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:47.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:47.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:47.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:47.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:47.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:54:47.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:47.957704+0000) 2019-10-18T04:54:48.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:48.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:48.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:48.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:49.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:49.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:49.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:49.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:50.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:50.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:50.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:50.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:51.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:51.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:51.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:51.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:52.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:52.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:52.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:52.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:52.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:52.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:52.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:54:52.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:52.947953+0000) 2019-10-18T04:54:53.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:54:53.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:53.758976+0000) 2019-10-18T04:54:53.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:54:53.790+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:53.795299+0000) 2019-10-18T04:54:53.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:53.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:53.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:53.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:54.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:54.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:54.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:54.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:55.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:55.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:55.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:55.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:56.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:56.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:56.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:56.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:57.755 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:54:57.750+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:54:57.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:57.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:57.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:57.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:57.957 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:54:57.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:54:57.957845+0000) 2019-10-18T04:54:58.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:58.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:58.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:58.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:54:59.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:59.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:54:59.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:54:59.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:00.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:00.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:00.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:00.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:01.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:01.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:01.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:01.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:02.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:02.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:02.947 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:55:02.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:02.948117+0000) 2019-10-18T04:55:02.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:02.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:02.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:02.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:03.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:55:03.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:03.759071+0000) 2019-10-18T04:55:03.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:55:03.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:03.795394+0000) 2019-10-18T04:55:04.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:04.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:04.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:04.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:04.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:04.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:04.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:04.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:06.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:06.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:06.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:06.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:07.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:07.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:07.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:07.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:07.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:07.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:07.957 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:55:07.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:07.957985+0000) 2019-10-18T04:55:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:08.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:08.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:09.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:09.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:09.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:09.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:10.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:10.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:10.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:10.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:10.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:10.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:10.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:10.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:11.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:11.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:11.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:11.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:12.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:12.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:12.947 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:55:12.943+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:12.948305+0000) 2019-10-18T04:55:12.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:12.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:12.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:12.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:13.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:55:13.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:13.759171+0000) 2019-10-18T04:55:13.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:55:13.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:13.795487+0000) 2019-10-18T04:55:13.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:13.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:13.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:13.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:14.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:14.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:14.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:14.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:15.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:15.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:15.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:15.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:16.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:16.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:16.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:16.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:17.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:17.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:17.961 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:55:17.951+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:17.958144+0000) 2019-10-18T04:55:18.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:18.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:18.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:18.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:19.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:18.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:19.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:18.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:19.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:19.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:19.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:19.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:21.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:21.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:21.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:21.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:22.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:22.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:22.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:22.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:22.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:22.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:22.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:55:22.942+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:22.948497+0000) 2019-10-18T04:55:23.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:23.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:23.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:23.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:23.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:55:23.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:23.759265+0000) 2019-10-18T04:55:23.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:55:23.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:23.795587+0000) 2019-10-18T04:55:24.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:24.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:24.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:24.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:25.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:25.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:25.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:25.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:26.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:26.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:26.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:27.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:27.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:27.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:27.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:27.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:27.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:27.957 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:55:27.954+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:27.958341+0000) 2019-10-18T04:55:28.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:28.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:28.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:28.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:29.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:29.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:29.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:29.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:30.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:30.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:30.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:30.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:30.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:30.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:30.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:30.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:31.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:31.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:31.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:31.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:32.761 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:32.757+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:32.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:55:32.942+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:32.948656+0000) 2019-10-18T04:55:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:33.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:33.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:33.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:33.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:55:33.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:33.759361+0000) 2019-10-18T04:55:33.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:55:33.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:33.795686+0000) 2019-10-18T04:55:33.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:33.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:33.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:33.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:34.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:34.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:34.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:34.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:35.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:35.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:35.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:35.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:36.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:36.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:36.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:36.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:37.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:37.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:37.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:37.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:37.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:37.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:37.957 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:55:37.954+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:37.958480+0000) 2019-10-18T04:55:38.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:38.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:38.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:38.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:39.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:39.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:39.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:39.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:40.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:40.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:40.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:40.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:41.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:41.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:41.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:41.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:42.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:42.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:42.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:55:42.942+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:42.948855+0000) 2019-10-18T04:55:42.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:42.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:42.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:42.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:43.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:55:43.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:43.759457+0000) 2019-10-18T04:55:43.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:55:43.792+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:43.795787+0000) 2019-10-18T04:55:43.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:43.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:43.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:43.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:44.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:44.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:44.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:44.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:45.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:45.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:45.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:45.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:46.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:46.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:46.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:46.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:47.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:47.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:47.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:47.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:47.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:47.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:47.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:55:47.954+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:47.958618+0000) 2019-10-18T04:55:48.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:48.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:48.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:48.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:49.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:49.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:49.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:49.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:50.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:50.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:50.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:50.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:51.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:51.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:51.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:51.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:52.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:52.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:52.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:52.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:52.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:55:52.942+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:52.949037+0000) 2019-10-18T04:55:53.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:55:53.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:53.759553+0000) 2019-10-18T04:55:53.795 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:55:53.792+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:53.795892+0000) 2019-10-18T04:55:53.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:53.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:53.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:53.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:54.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:54.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:54.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:54.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:55.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:55.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:55.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:55.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:56.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:56.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:56.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:56.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:57.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:55:57.760+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:55:57.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:57.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:57.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:57.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:57.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:55:57.954+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:55:57.958778+0000) 2019-10-18T04:55:58.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:58.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:58.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:58.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:55:59.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:59.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:55:59.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:55:59.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:00.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:00.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:00.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:00.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:01.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:01.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:01.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:01.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:02.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:02.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:02.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:02.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:02.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:02.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:02.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:02.942+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:02.949214+0000) 2019-10-18T04:56:03.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:56:03.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:03.759649+0000) 2019-10-18T04:56:03.796 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:56:03.792+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:03.795991+0000) 2019-10-18T04:56:03.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:03.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:03.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:03.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:04.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:04.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:04.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:04.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:05.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:05.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:05.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:05.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:06.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:06.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:06.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:06.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:07.767 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:07.764+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:07.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:07.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:07.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:07.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:07.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:56:07.954+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:07.958927+0000) 2019-10-18T04:56:08.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:08.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:08.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:08.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:09.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:09.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:10.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:10.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:10.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:10.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:11.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:11.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:11.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:11.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:12.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:12.763+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:12.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:12.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:12.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:12.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:12.948 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:12.942+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:12.949394+0000) 2019-10-18T04:56:13.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:13.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:13.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:13.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:13.759 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:56:13.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:13.759746+0000) 2019-10-18T04:56:13.796 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:56:13.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:13.796089+0000) 2019-10-18T04:56:14.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:14.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:14.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:14.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:15.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:15.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:15.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:15.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:16.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:16.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:16.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:16.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:17.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:17.763+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:17.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:17.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:17.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:17.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:17.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:56:17.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:17.959110+0000) 2019-10-18T04:56:18.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:18.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:18.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:18.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:19.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:19.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:19.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:19.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:20.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:20.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:20.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:20.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:21.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:21.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:21.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:21.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:22.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:22.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:22.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:22.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:22.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:22.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:22.949 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:22.946+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:22.949565+0000) 2019-10-18T04:56:23.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:56:23.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:23.759845+0000) 2019-10-18T04:56:23.796 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:56:23.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:23.796192+0000) 2019-10-18T04:56:23.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:23.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:23.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:23.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:24.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:24.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:24.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:24.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:25.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:25.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:25.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:25.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:26.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:26.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:26.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:26.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:27.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:27.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:27.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:27.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:27.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:27.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:27.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:56:27.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:27.959257+0000) 2019-10-18T04:56:28.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:28.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:28.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:28.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:29.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:29.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:29.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:30.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:30.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:30.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:30.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:31.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:31.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:31.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:31.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:32.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:32.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:32.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:32.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:32.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:32.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:32.949 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:32.945+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:32.949725+0000) 2019-10-18T04:56:33.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:56:33.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:33.759942+0000) 2019-10-18T04:56:33.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:56:33.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:33.796289+0000) 2019-10-18T04:56:33.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:33.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:33.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:33.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:34.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:34.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:34.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:34.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:35.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:35.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:35.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:35.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:36.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:36.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:36.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:36.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:37.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:37.767+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:37.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:37.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:37.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:37.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:37.958 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:56:37.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:37.959409+0000) 2019-10-18T04:56:38.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:38.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:38.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:38.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:39.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:39.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:40.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:40.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:40.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:40.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:41.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:41.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:41.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:41.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:42.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:42.771+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:42.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:42.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:42.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:42.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:42.949 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:42.945+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:42.949856+0000) 2019-10-18T04:56:43.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:56:43.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:43.760043+0000) 2019-10-18T04:56:43.796 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:56:43.791+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:43.796391+0000) 2019-10-18T04:56:43.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:43.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:43.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:43.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:44.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:44.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:44.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:44.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:45.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:45.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:45.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:45.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:46.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:46.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:46.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:46.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:47.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:47.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:47.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:47.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:47.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:47.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:47.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:56:47.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:47.959609+0000) 2019-10-18T04:56:48.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:48.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:48.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:48.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:49.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:49.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:49.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:49.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:50.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:50.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:50.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:50.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:51.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:51.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:51.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:51.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:52.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:52.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:52.950 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:52.945+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:52.949989+0000) 2019-10-18T04:56:52.950 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:52.949+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:52.951080+0000) 2019-10-18T04:56:52.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:52.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:52.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:52.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:53.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:56:53.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:53.760140+0000) 2019-10-18T04:56:53.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:56:53.790+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:53.796494+0000) 2019-10-18T04:56:53.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:53.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:53.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:53.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 432 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:54.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:54.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:54.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:54.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:56.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:56.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:56.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:56.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:57.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:57.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:57.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:57.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:57.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:56:57.449+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:57.450452+0000) 2019-10-18T04:56:57.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:56:57.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:56:57.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:56:57.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:56:57.959769+0000) 2019-10-18T04:56:58.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:58.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:58.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:58.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:56:59.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:59.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:56:59.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:56:59.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:00.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:00.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:46:28.130823+0000 (oldest deadline 2019-10-18T04:46:48.130823+0000) 2019-10-18T04:57:00.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:00.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:01.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:01.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:02.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:02.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:02.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:02.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:03.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:03.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:03.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:57:03.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:03.760237+0000) 2019-10-18T04:57:03.796 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:57:03.790+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:03.796591+0000) 2019-10-18T04:57:04.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:04.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:05.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:05.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:06.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:06.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:07.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:07.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:07.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:57:07.449+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:07.450633+0000) 2019-10-18T04:57:07.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:07.770+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:07.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:57:07.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:07.959953+0000) 2019-10-18T04:57:08.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:08.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:09.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:09.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:10.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:10.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:11.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:11.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:12.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:12.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:12.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:12.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:13.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:13.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:13.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:57:13.754+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:13.760334+0000) 2019-10-18T04:57:13.796 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:57:13.790+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:13.796690+0000) 2019-10-18T04:57:14.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:14.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:15.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:15.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:16.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:16.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:17.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:17.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:17.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:57:17.449+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:17.450812+0000) 2019-10-18T04:57:17.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:17.774+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:17.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:57:17.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:17.960139+0000) 2019-10-18T04:57:18.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:18.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:19.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:19.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:20.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:20.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:21.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:21.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:22.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:22.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:22.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:22.773+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:23.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:23.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:23.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:57:23.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:23.760431+0000) 2019-10-18T04:57:23.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:57:23.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:23.796789+0000) 2019-10-18T04:57:24.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:24.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:25.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:25.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:26.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:26.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:27.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:27.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:27.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:27.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:27.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:57:27.449+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:27.451022+0000) 2019-10-18T04:57:27.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:27.773+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:27.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:57:27.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:27.960290+0000) 2019-10-18T04:57:28.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:28.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:28.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:28.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:29.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:29.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:29.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:29.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:30.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:30.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:30.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:30.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:31.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:31.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:31.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:31.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:32.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:32.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:32.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:32.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:32.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:32.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:33.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:33.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:33.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:33.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:33.760 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:57:33.753+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:33.760527+0000) 2019-10-18T04:57:33.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:57:33.789+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:33.796886+0000) 2019-10-18T04:57:34.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:34.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:34.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:34.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:35.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:35.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:35.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:35.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:36.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:36.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:36.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:36.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:37.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:37.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:37.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:37.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:37.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:57:37.449+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:37.451235+0000) 2019-10-18T04:57:37.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:37.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:37.959 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:57:37.957+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:37.960464+0000) 2019-10-18T04:57:38.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:38.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:38.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:38.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:39.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:39.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:39.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:39.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:40.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:40.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:40.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:40.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:41.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:41.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:41.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:41.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:42.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:42.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:42.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:42.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:42.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:42.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:43.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:43.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:43.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:43.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:43.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:57:43.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:43.760630+0000) 2019-10-18T04:57:43.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:57:43.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:43.797271+0000) 2019-10-18T04:57:44.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:44.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:44.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:44.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:45.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:45.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:45.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:45.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:46.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:46.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:46.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:46.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:47.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:47.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:47.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:47.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:47.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:57:47.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:47.451444+0000) 2019-10-18T04:57:47.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:47.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:47.960 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:57:47.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:47.960649+0000) 2019-10-18T04:57:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:48.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:48.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:48.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:49.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:49.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:49.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:49.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:50.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:50.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:50.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:50.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:51.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:51.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:51.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:51.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:52.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:52.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:52.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:52.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:52.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:52.781+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:53.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:53.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:53.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:53.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:53.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:57:53.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:53.760733+0000) 2019-10-18T04:57:53.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:57:53.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:53.797373+0000) 2019-10-18T04:57:54.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:54.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:54.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:54.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:55.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:55.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:55.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:55.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:56.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:56.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:57.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:57.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:57.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:57.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:57.451 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:57:57.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:57.451626+0000) 2019-10-18T04:57:57.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:57:57.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:57:57.962 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:57:57.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:57:57.960832+0000) 2019-10-18T04:57:58.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:58.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:58.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:58.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:57:59.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:59.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:57:59.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:57:59.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:00.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:00.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:00.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:00.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:01.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:01.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:01.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:01.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:02.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:02.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:02.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:02.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:02.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:02.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:03.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:03.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:03.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:03.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:03.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:58:03.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:03.760831+0000) 2019-10-18T04:58:03.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:58:03.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:03.797466+0000) 2019-10-18T04:58:04.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:04.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:04.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:04.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:05.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:05.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:05.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:05.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:06.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:06.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:06.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:06.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:07.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:07.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:07.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:07.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:07.451 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:58:07.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:07.451804+0000) 2019-10-18T04:58:07.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:07.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:07.960 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:58:07.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:07.961043+0000) 2019-10-18T04:58:08.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:08.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:08.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:08.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:09.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:09.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:09.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:09.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:10.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:10.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:10.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:10.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:11.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:11.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:11.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:11.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:12.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:12.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:12.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:12.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:12.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:12.784+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:13.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:13.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:13.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:13.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:13.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:58:13.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:13.760929+0000) 2019-10-18T04:58:13.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:58:13.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:13.797564+0000) 2019-10-18T04:58:14.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:14.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:14.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:14.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:15.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:15.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:15.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:15.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:16.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:16.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:16.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:16.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:17.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:17.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:17.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:17.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:17.451 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:58:17.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:17.451993+0000) 2019-10-18T04:58:17.790 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:17.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:17.962 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:58:17.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:17.961220+0000) 2019-10-18T04:58:18.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:18.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:18.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:18.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:19.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:19.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:19.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:19.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:20.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:20.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:20.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:20.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:21.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:21.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:21.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:21.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:22.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:22.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:22.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:22.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:22.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:22.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:23.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:23.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:23.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:23.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:58:23.756+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:23.761036+0000) 2019-10-18T04:58:23.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:58:23.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:23.797665+0000) 2019-10-18T04:58:24.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:24.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:24.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:24.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:25.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:25.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:25.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:26.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:26.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:26.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:26.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:27.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:27.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:27.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:27.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:27.451 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:58:27.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:27.452186+0000) 2019-10-18T04:58:27.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:27.788+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:27.960 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:58:27.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:27.961405+0000) 2019-10-18T04:58:28.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:28.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:28.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:28.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:29.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:29.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:29.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:29.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:30.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:30.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:30.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:30.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:31.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:31.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:31.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:31.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:32.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:32.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:32.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:32.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:32.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:32.787+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:33.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:33.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:33.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:33.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:33.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:58:33.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:33.761131+0000) 2019-10-18T04:58:33.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:58:33.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:33.797761+0000) 2019-10-18T04:58:34.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:34.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:34.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:34.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:35.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:35.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:35.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:35.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:36.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:36.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:36.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:36.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:37.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:37.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:37.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:37.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:37.452 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:58:37.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:37.452400+0000) 2019-10-18T04:58:37.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:37.787+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:37.961 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:58:37.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:37.961611+0000) 2019-10-18T04:58:38.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:38.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:38.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:38.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:39.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:39.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:39.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:39.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:40.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:40.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:40.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:40.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:41.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:41.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:41.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:41.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:42.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:42.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:42.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:42.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:42.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:42.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:43.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:43.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:43.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:43.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:43.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:58:43.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:43.761233+0000) 2019-10-18T04:58:43.797 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:58:43.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:43.797855+0000) 2019-10-18T04:58:44.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:44.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:44.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:44.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:45.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:45.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:45.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:45.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:46.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:46.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:46.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:46.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:47.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:47.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:47.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:47.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:47.452 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:58:47.448+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:47.452582+0000) 2019-10-18T04:58:47.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:47.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:47.961 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:58:47.956+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:47.961798+0000) 2019-10-18T04:58:48.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:48.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:48.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:48.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:49.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:49.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:49.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:49.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:50.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:50.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:50.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:50.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:51.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:51.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:51.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:51.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:52.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:52.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:52.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:52.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:52.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:52.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:53.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:53.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:53.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:53.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:53.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:58:53.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:53.761329+0000) 2019-10-18T04:58:53.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:58:53.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:53.797949+0000) 2019-10-18T04:58:54.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:54.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:54.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:54.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:55.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:55.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:55.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:55.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:56.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:56.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:56.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:56.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:57.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:57.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:57.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:57.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:57.456 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:58:57.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:57.452767+0000) 2019-10-18T04:58:57.797 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:58:57.791+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:58:57.964 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:58:57.955+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:58:57.961980+0000) 2019-10-18T04:58:58.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:58.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:58.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:58.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:58:59.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:59.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:58:59.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:58:59.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:00.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:00.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:00.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:00.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:01.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:01.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:01.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:01.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:02.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:02.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:02.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:02.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:02.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:02.795+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:03.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:03.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:03.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:03.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:03.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:59:03.755+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:03.761426+0000) 2019-10-18T04:59:03.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:59:03.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:03.798045+0000) 2019-10-18T04:59:04.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:04.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:04.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:04.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:05.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:05.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:05.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:05.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:06.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:06.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:06.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:06.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:07.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:07.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:07.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:07.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:07.452 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:59:07.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:07.452977+0000) 2019-10-18T04:59:07.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:07.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:07.961 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:59:07.955+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:07.962155+0000) 2019-10-18T04:59:08.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:08.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:08.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:08.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:09.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:09.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:09.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:10.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:10.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:10.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:10.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:11.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:11.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:11.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:11.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:12.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:12.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:12.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:12.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:12.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:12.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:13.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:13.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:13.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:13.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:13.761 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:59:13.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:13.761534+0000) 2019-10-18T04:59:13.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:59:13.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:13.798141+0000) 2019-10-18T04:59:14.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:14.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:14.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:14.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:15.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:15.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:15.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:15.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:16.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:16.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:16.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:16.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:17.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:17.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:17.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:17.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:17.452 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:59:17.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:17.453167+0000) 2019-10-18T04:59:17.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:17.794+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:17.961 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:59:17.959+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:17.962360+0000) 2019-10-18T04:59:18.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:18.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:18.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:18.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:19.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:19.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:19.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:19.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:20.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:20.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:20.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:21.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:21.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:21.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:21.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:22.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:22.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:22.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:22.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:22.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:22.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:23.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:23.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:23.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:23.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:23.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:59:23.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:23.761628+0000) 2019-10-18T04:59:23.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:59:23.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:23.798242+0000) 2019-10-18T04:59:24.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:24.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:24.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:24.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:25.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:25.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:25.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:25.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:26.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:26.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:26.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:26.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:27.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:27.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:27.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:27.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:27.452 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:59:27.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:27.453357+0000) 2019-10-18T04:59:27.803 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:27.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:27.962 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:59:27.959+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:27.962547+0000) 2019-10-18T04:59:28.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:28.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:28.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:28.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:29.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:29.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:30.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:30.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:30.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:30.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:31.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:31.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:31.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:31.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:32.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:32.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:32.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:32.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:32.803 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:32.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:33.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:33.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:33.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:33.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:33.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:59:33.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:33.761734+0000) 2019-10-18T04:59:33.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:59:33.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:33.798338+0000) 2019-10-18T04:59:34.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:34.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:34.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:34.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:35.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:35.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:35.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:35.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:36.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:36.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:36.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:36.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:37.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:37.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:37.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:37.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:37.453 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:59:37.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:37.453546+0000) 2019-10-18T04:59:37.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:37.798+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:37.962 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:59:37.959+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:37.962732+0000) 2019-10-18T04:59:38.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:38.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:38.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:39.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:39.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:39.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:39.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:40.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:40.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:41.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:41.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:41.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:41.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:42.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:42.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:42.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:42.797+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:43.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:43.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:43.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:43.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:43.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:59:43.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:43.761836+0000) 2019-10-18T04:59:43.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:59:43.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:43.798438+0000) 2019-10-18T04:59:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:44.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:44.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:45.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:45.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:45.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:45.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:46.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:46.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:46.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:46.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:47.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:47.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:47.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:47.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:47.453 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:59:47.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:47.453733+0000) 2019-10-18T04:59:47.808 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:47.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:47.963 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:59:47.959+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:47.962911+0000) 2019-10-18T04:59:48.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:48.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:48.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:48.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:49.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:49.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:49.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:49.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:50.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:50.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:50.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:50.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:51.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:51.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:51.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:51.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:52.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:52.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:52.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:52.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:52.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:52.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:53.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:53.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:53.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:53.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:53.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T04:59:53.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:53.761936+0000) 2019-10-18T04:59:53.798 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T04:59:53.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:53.798533+0000) 2019-10-18T04:59:54.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:54.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:54.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:54.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:55.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:55.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:55.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:55.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:56.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:56.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:56.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:56.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:57.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:57.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:57.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:57.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:57.453 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T04:59:57.447+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:57.453919+0000) 2019-10-18T04:59:57.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T04:59:57.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T04:59:57.962 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T04:59:57.959+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T03:59:57.963090+0000) 2019-10-18T04:59:58.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:58.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:58.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:58.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T04:59:59.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:59.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T04:59:59.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T04:59:59.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:00.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:00.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:00.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:00.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:01.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:01.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:01.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:01.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:02.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:02.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:02.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:02.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:02.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:02.805+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:03.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:03.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:03.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:03.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:03.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:00:03.761+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:03.762030+0000) 2019-10-18T05:00:03.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:00:03.797+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:03.798633+0000) 2019-10-18T05:00:04.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:04.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:04.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:04.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:05.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:05.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:05.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:05.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:06.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:06.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:06.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:06.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:07.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:07.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:07.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:07.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:07.454 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:07.446+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:07.454096+0000) 2019-10-18T05:00:07.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:07.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:07.962 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:00:07.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:07.963282+0000) 2019-10-18T05:00:08.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:08.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:08.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:08.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:09.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:09.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:09.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:09.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:10.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:10.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:10.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:10.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:11.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:11.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:11.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:11.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:12.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:12.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:12.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:12.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:12.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:12.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:13.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:13.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:13.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:13.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:13.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:00:13.761+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:13.762127+0000) 2019-10-18T05:00:13.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:00:13.797+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:13.798729+0000) 2019-10-18T05:00:14.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:14.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:14.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:14.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:15.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:15.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:15.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:16.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:16.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:16.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:16.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:17.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:17.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:17.453 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:17.450+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:17.454270+0000) 2019-10-18T05:00:17.454 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:17.450+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:17.455148+0000) 2019-10-18T05:00:17.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:17.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:17.964 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:00:17.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:17.963461+0000) 2019-10-18T05:00:18.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:18.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:18.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:18.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:19.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:19.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:19.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:19.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:20.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:20.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:20.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:20.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:21.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:21.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:21.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:21.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:21.954 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:21.950+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:21.954660+0000) 2019-10-18T05:00:22.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:22.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:22.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:22.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:22.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:22.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:23.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:23.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:23.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:23.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:23.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:00:23.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:23.762227+0000) 2019-10-18T05:00:23.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:00:23.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:23.798827+0000) 2019-10-18T05:00:24.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:24.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:24.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:24.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:25.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:25.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:25.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:25.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:26.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:26.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:26.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:26.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:27.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:27.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:27.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:27.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:27.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:27.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:27.963 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:00:27.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:27.963644+0000) 2019-10-18T05:00:28.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:28.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:28.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:28.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:29.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:29.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:29.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:29.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:30.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:30.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:30.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:30.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:31.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:31.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:31.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:31.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:31.954 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:31.950+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:31.954803+0000) 2019-10-18T05:00:32.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:32.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:32.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:32.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:32.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:32.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:33.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:33.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:33.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:33.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:33.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:00:33.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:33.762321+0000) 2019-10-18T05:00:33.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:00:33.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:33.798924+0000) 2019-10-18T05:00:34.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:34.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:34.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:34.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:35.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:35.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:35.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:35.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:36.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:36.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:36.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:36.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:37.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:37.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:37.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:37.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:37.814 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:37.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:37.963 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:00:37.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:37.963855+0000) 2019-10-18T05:00:38.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:38.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:38.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:38.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:39.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:39.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:39.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:39.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:40.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:40.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:40.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:40.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:41.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:41.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:41.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:41.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:41.954 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:41.950+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:41.955016+0000) 2019-10-18T05:00:42.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:42.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:42.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:42.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:42.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:42.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:43.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:43.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:43.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:43.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:43.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:00:43.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:43.762420+0000) 2019-10-18T05:00:43.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:00:43.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:43.799020+0000) 2019-10-18T05:00:44.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:44.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:44.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:44.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:45.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:45.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:45.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:45.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:46.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:46.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:46.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:46.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:47.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:47.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:47.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:47.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:47.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:47.812+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:47.963 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:00:47.958+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:47.964036+0000) 2019-10-18T05:00:48.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:48.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:48.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:48.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:49.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:49.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:49.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:49.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:50.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:50.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:50.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:50.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:51.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:51.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:51.954 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:00:51.950+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:51.955194+0000) 2019-10-18T05:00:52.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:52.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:52.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:52.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:52.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:52.811+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:53.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:53.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:53.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:53.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:53.762 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:00:53.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:53.762518+0000) 2019-10-18T05:00:53.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:00:53.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:53.799128+0000) 2019-10-18T05:00:54.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:54.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:54.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:54.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:55.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:55.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:55.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:55.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:56.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:56.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:56.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:56.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:57.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:57.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:57.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:57.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:57.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:00:57.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:00:57.963 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:00:57.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:00:57.964221+0000) 2019-10-18T05:00:58.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:58.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:58.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:58.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:00:59.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:59.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:00:59.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:00:59.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:00.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:00.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:00.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:00.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:01.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:01.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:01.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:01.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:01.955 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:01:01.954+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:01.955343+0000) 2019-10-18T05:01:02.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:02.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:02.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:02.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:02.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:02.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:03.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:03.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:03.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:03.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:03.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:01:03.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:03.762612+0000) 2019-10-18T05:01:03.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:01:03.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:03.799228+0000) 2019-10-18T05:01:04.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:04.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:04.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:04.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:05.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:05.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:05.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:05.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:06.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:06.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:06.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:06.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:07.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:07.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:07.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:07.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:07.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:07.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:07.964 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:01:07.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:07.964431+0000) 2019-10-18T05:01:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:08.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:08.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:09.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:09.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:09.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:09.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:10.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:10.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:10.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:10.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:11.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:11.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:11.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:11.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:11.955 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:01:11.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:11.955528+0000) 2019-10-18T05:01:12.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:12.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:12.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:12.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:12.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:12.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:13.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:13.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:13.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:13.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:13.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:01:13.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:13.762714+0000) 2019-10-18T05:01:13.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:01:13.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:13.799323+0000) 2019-10-18T05:01:14.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:14.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:14.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:14.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:15.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:15.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:15.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:15.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:16.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:16.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:16.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:16.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:17.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:17.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:17.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:17.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:17.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:17.815+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:17.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:01:17.961+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:17.964639+0000) 2019-10-18T05:01:18.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:18.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:18.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:18.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:19.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:19.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:19.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:19.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:20.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:20.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:20.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:20.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:21.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:21.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:21.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:21.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:21.955 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:01:21.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:21.955711+0000) 2019-10-18T05:01:22.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:22.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:22.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:22.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:22.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:22.819+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:23.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:23.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:23.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:23.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:23.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:01:23.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:23.762817+0000) 2019-10-18T05:01:23.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:01:23.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:23.799426+0000) 2019-10-18T05:01:24.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:24.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:24.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:24.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:25.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:25.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:25.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:25.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:26.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:26.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:26.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:26.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:27.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:27.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:27.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:27.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:27.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:27.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:27.964 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:01:27.961+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:27.964821+0000) 2019-10-18T05:01:28.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:28.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:28.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:28.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:29.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:29.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:29.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:29.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:30.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:30.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:30.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:30.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:31.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:31.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:31.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:31.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:31.955 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:01:31.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:31.955894+0000) 2019-10-18T05:01:32.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:32.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:32.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:32.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:32.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:32.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:33.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:33.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:33.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:33.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:33.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:01:33.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:33.762912+0000) 2019-10-18T05:01:33.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:01:33.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:33.799525+0000) 2019-10-18T05:01:34.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:34.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:34.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:34.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:35.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:35.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:35.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:35.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:36.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:36.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:36.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:36.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:37.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:37.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:37.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:37.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:37.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:37.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:37.964 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:01:37.961+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:37.965001+0000) 2019-10-18T05:01:38.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:38.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:38.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:38.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:39.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:39.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:39.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:39.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:40.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:40.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:40.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:40.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:41.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:41.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:41.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:41.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:41.955 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:01:41.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:41.956070+0000) 2019-10-18T05:01:42.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:42.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:42.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:42.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:42.826 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:42.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:43.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:43.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:43.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:43.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:43.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:01:43.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:43.763010+0000) 2019-10-18T05:01:43.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:01:43.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:43.799625+0000) 2019-10-18T05:01:44.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:44.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:44.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:44.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:45.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:45.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:46.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:46.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:46.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:46.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:47.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:47.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:47.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:47.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:47.826 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:47.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:47.965 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:01:47.961+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:47.965188+0000) 2019-10-18T05:01:48.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:48.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:48.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:48.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:49.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:49.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:49.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:49.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:50.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:50.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:50.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:50.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:51.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:51.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:51.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:51.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:51.955 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:01:51.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:51.956251+0000) 2019-10-18T05:01:52.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:52.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:52.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:52.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:52.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:52.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:53.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:53.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:53.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:53.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:53.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:01:53.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:53.763110+0000) 2019-10-18T05:01:53.799 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:01:53.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:53.799723+0000) 2019-10-18T05:01:54.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:54.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:54.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:54.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:55.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:55.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:55.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:55.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:56.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:56.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:56.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:56.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:57.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:57.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:57.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:57.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:57.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:01:57.822+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:01:57.965 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:01:57.961+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:01:57.965406+0000) 2019-10-18T05:01:58.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:58.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:58.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:58.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:01:59.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:59.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:01:59.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:01:59.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:00.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:00.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:00.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:00.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:01.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:01.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:01.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:01.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:01.956 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:02:01.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:01.956438+0000) 2019-10-18T05:02:02.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:02.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:02.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:02.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:02.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:02.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:03.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:03.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:03.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:03.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:03.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:02:03.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:03.763203+0000) 2019-10-18T05:02:03.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:02:03.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:03.799823+0000) 2019-10-18T05:02:04.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:04.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:04.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:04.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:05.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:05.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:05.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:05.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:06.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:06.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:06.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:06.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:07.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:07.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:07.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:07.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:07.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:07.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:07.965 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:02:07.961+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:07.965622+0000) 2019-10-18T05:02:08.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:08.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:08.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:08.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:09.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:09.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:09.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:09.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:10.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:10.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:10.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:10.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:11.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:11.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:11.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:11.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:11.956 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:02:11.953+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:11.956630+0000) 2019-10-18T05:02:12.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:12.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:12.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:12.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:12.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:12.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:13.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:13.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:13.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:13.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:13.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:02:13.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:13.763303+0000) 2019-10-18T05:02:13.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:02:13.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:13.799923+0000) 2019-10-18T05:02:14.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:14.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:14.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:14.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:15.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:15.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:15.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:15.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:16.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:16.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:16.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:16.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:17.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:17.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:17.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:17.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:17.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:17.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:17.965 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:02:17.960+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:17.965832+0000) 2019-10-18T05:02:18.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:18.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:18.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:18.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:19.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:19.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:19.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:19.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:20.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:20.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:20.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:20.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:21.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:21.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:21.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:21.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:21.956 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:02:21.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:21.956790+0000) 2019-10-18T05:02:22.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:22.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:22.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:22.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:22.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:22.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:23.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:23.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:23.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:23.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:23.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:02:23.761+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:23.763400+0000) 2019-10-18T05:02:23.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:02:23.797+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:23.800020+0000) 2019-10-18T05:02:24.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:24.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:24.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:24.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:25.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:25.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:25.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:25.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:26.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:26.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:26.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:26.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:27.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:27.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:27.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:27.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:27.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:27.829+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:27.965 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:02:27.960+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:27.966041+0000) 2019-10-18T05:02:28.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:28.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:28.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:28.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:29.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:29.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:29.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:29.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:30.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:30.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:30.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:30.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:31.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:31.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:31.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:31.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:31.956 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:02:31.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:31.956943+0000) 2019-10-18T05:02:32.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:32.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:32.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:32.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:32.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:32.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:33.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:33.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:33.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:33.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:33.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:02:33.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:33.763493+0000) 2019-10-18T05:02:33.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:02:33.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:33.800119+0000) 2019-10-18T05:02:34.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:34.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:34.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:34.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:35.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:35.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:35.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:35.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:36.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:36.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:36.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:36.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:37.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:37.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:37.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:37.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:37.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:37.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:37.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:02:37.964+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:37.966219+0000) 2019-10-18T05:02:38.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:38.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:38.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:38.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:39.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:39.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:39.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:39.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:40.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:40.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:40.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:40.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:41.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:41.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:41.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:41.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:41.958 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:02:41.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:41.957127+0000) 2019-10-18T05:02:42.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:42.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:42.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:42.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:42.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:42.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:43.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:43.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:43.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:43.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:02:43.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:43.763592+0000) 2019-10-18T05:02:43.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:02:43.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:43.800219+0000) 2019-10-18T05:02:44.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:44.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:44.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:44.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:45.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:45.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:45.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:45.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:46.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:46.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:46.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:46.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:47.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:47.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:47.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:47.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:47.966 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:02:47.964+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:47.966398+0000) 2019-10-18T05:02:48.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:48.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:48.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:48.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:49.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:49.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:49.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:49.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:50.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:50.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:50.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:50.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:51.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:51.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:51.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:51.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:51.956 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:02:51.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:51.957305+0000) 2019-10-18T05:02:52.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:52.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:52.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:52.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:52.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:52.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:53.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:53.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:53.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:53.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:53.763 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:02:53.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:53.763687+0000) 2019-10-18T05:02:53.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:02:53.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:53.800316+0000) 2019-10-18T05:02:54.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:54.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:54.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:54.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:55.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:55.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:55.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:55.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:56.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:56.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:56.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:56.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:57.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:57.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:57.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:57.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:57.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:02:57.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:02:57.966 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:02:57.964+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:02:57.966591+0000) 2019-10-18T05:02:58.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:58.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:58.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:58.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:02:59.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:59.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:02:59.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:02:59.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:00.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:00.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:00.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:00.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:01.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:01.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:01.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:01.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:01.957 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:03:01.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:01.957484+0000) 2019-10-18T05:03:02.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:02.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:02.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:02.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:02.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:02.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:03.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:03.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:03.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:03.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:03.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:03:03.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:03.763792+0000) 2019-10-18T05:03:03.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:03:03.796+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:03.800416+0000) 2019-10-18T05:03:04.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:04.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:04.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:04.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:05.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:05.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:05.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:05.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:06.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:06.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:06.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:06.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:07.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:07.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:07.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:07.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:07.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:07.836+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:07.966 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:03:07.964+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:07.966755+0000) 2019-10-18T05:03:08.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:08.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:08.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:08.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:09.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:09.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:09.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:09.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:10.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:10.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:10.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:10.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:11.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:11.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:11.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:11.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:11.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:03:11.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:11.957630+0000) 2019-10-18T05:03:12.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:12.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:12.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:12.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:12.845 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:12.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:13.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:13.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:13.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:13.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:13.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:03:13.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:13.763893+0000) 2019-10-18T05:03:13.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:03:13.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:13.800528+0000) 2019-10-18T05:03:14.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:14.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:14.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:14.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:15.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:15.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:15.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:15.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:16.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:16.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:16.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:16.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:17.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:17.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:17.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:17.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:17.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:17.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:17.966 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:03:17.964+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:17.966947+0000) 2019-10-18T05:03:18.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:18.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:18.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:18.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:19.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:19.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:19.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:19.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:20.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:20.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:20.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:20.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:21.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:21.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:21.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:21.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:21.957 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:03:21.952+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:21.957818+0000) 2019-10-18T05:03:22.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:22.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:22.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:22.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:22.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:22.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:23.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:23.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:23.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:23.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:23.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:03:23.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:23.763990+0000) 2019-10-18T05:03:23.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:03:23.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:23.800627+0000) 2019-10-18T05:03:24.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:24.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:24.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:24.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:25.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:25.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:25.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:25.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:26.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:26.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:26.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:26.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:27.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:27.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:27.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:27.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:27.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:27.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:27.966 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:03:27.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:27.967165+0000) 2019-10-18T05:03:28.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:28.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:28.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:28.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:29.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:29.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:29.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:29.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:30.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:30.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:30.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:30.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:31.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:31.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:31.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:31.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:31.957 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:03:31.951+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:31.958008+0000) 2019-10-18T05:03:32.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:32.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:32.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:32.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:32.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:32.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:33.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:33.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:33.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:33.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:33.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:03:33.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:33.764085+0000) 2019-10-18T05:03:33.800 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:03:33.795+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:33.800726+0000) 2019-10-18T05:03:34.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:34.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:34.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:34.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:35.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:35.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:35.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:35.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:36.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:36.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:36.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:36.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:37.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:37.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:37.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:37.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:37.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:37.839+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:37.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:03:37.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:37.967363+0000) 2019-10-18T05:03:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:38.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:38.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:39.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:39.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:39.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:39.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:40.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:40.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:40.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:40.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:41.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:41.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:41.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:41.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:41.957 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:03:41.955+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:41.958213+0000) 2019-10-18T05:03:42.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:42.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:42.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:42.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:42.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:42.843+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:43.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:43.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:43.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:43.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:43.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:03:43.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:43.764191+0000) 2019-10-18T05:03:43.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:03:43.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:43.800821+0000) 2019-10-18T05:03:44.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:44.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:44.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:44.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:45.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:45.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:45.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:45.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:46.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:46.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:46.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:46.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:47.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:47.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:47.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:47.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:47.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:47.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:47.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:03:47.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:47.967546+0000) 2019-10-18T05:03:48.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:48.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:48.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:48.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:49.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:49.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:49.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:49.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:50.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:50.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:50.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:50.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:51.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:51.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:51.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:51.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:51.970 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:03:51.955+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:51.958421+0000) 2019-10-18T05:03:52.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:52.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:52.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:52.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:52.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:52.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:53.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:53.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:53.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:53.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:53.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:03:53.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:53.764289+0000) 2019-10-18T05:03:53.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:03:53.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:53.800920+0000) 2019-10-18T05:03:54.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:54.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:54.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:54.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:55.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:55.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:55.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:55.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:56.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:56.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:56.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:56.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:57.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:57.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:57.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:57.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:57.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:03:57.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:03:57.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:03:57.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:03:57.967738+0000) 2019-10-18T05:03:58.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:58.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:58.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:58.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:03:59.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:59.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:03:59.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:03:59.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:00.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:00.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:00.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:00.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:01.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:01.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:01.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:01.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:01.959 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:04:01.955+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:01.958605+0000) 2019-10-18T05:04:02.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:02.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:02.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:02.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:02.850 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:02.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:03.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:03.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:03.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:03.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:03.764 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:04:03.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:03.764384+0000) 2019-10-18T05:04:03.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:04:03.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:03.801019+0000) 2019-10-18T05:04:04.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:04.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:04.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:04.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:05.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:05.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:05.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:05.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:06.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:06.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:06.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:06.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:07.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:07.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:07.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:07.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:07.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:07.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:07.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:04:07.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:07.967911+0000) 2019-10-18T05:04:08.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:08.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:08.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:08.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:09.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:09.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:09.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:09.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:10.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:10.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:10.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:10.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:11.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:11.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:11.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:11.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:11.958 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:04:11.955+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:11.958788+0000) 2019-10-18T05:04:12.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:12.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:12.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:12.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:12.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:12.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:13.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:13.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:13.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:13.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:13.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:04:13.758+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:13.764497+0000) 2019-10-18T05:04:13.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:04:13.794+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:13.801118+0000) 2019-10-18T05:04:14.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:14.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:14.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:14.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:15.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:15.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:15.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:15.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:16.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:16.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:16.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:16.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:17.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:17.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:17.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:17.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:17.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:17.846+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:17.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:04:17.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:17.968091+0000) 2019-10-18T05:04:18.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:18.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:18.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:18.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:19.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:19.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:19.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:19.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:20.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:20.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:20.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:20.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:21.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:21.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:21.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:21.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:21.958 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:04:21.955+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:21.958936+0000) 2019-10-18T05:04:22.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:22.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:22.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:22.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:22.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:22.845+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:23.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:23.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:23.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:23.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:23.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:04:23.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:23.764593+0000) 2019-10-18T05:04:23.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:04:23.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:23.801213+0000) 2019-10-18T05:04:24.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:24.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:24.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:24.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:25.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:25.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:25.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:25.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:26.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:26.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:26.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:26.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:27.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:27.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:27.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:27.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:27.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:27.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:27.967 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:04:27.963+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:27.968261+0000) 2019-10-18T05:04:28.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:28.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:28.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:28.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:29.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:29.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:29.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:29.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:30.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:30.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:30.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:30.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:31.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:31.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:31.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:31.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:31.958 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:04:31.955+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:31.959075+0000) 2019-10-18T05:04:32.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:32.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:32.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:32.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:32.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:32.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:33.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:33.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:33.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:33.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:33.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:04:33.757+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:33.764688+0000) 2019-10-18T05:04:33.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:04:33.793+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:33.801312+0000) 2019-10-18T05:04:34.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:34.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:34.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:34.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:35.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:35.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:35.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:35.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:36.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:36.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:36.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:36.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:37.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:37.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:37.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:37.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:37.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:37.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:37.968 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:04:37.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:37.968438+0000) 2019-10-18T05:04:38.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:38.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:38.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:38.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:39.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:39.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:39.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:39.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:40.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:40.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:40.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:40.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:41.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:41.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:41.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:41.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:41.959 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:04:41.954+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:41.959263+0000) 2019-10-18T05:04:42.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:42.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:42.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:42.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:42.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:42.853+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:43.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:43.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:43.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:43.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:43.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:04:43.761+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:43.764787+0000) 2019-10-18T05:04:43.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:04:43.797+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:43.801412+0000) 2019-10-18T05:04:44.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:44.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:44.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:44.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:45.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:45.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:45.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:45.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:46.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:46.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:46.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:46.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:47.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:47.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:47.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:47.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:47.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:47.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:47.968 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:04:47.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:47.968619+0000) 2019-10-18T05:04:48.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:48.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:48.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:48.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:49.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:49.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:49.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:49.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:50.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:50.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:50.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:50.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:51.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:51.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:51.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:51.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:51.959 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:04:51.954+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:51.959435+0000) 2019-10-18T05:04:52.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:52.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:52.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:52.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:52.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:52.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:53.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:53.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:53.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:53.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:53.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:04:53.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:53.764882+0000) 2019-10-18T05:04:53.801 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:04:53.800+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:53.801518+0000) 2019-10-18T05:04:54.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:54.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:54.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:54.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:55.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:55.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:55.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:55.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:56.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:56.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:56.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:56.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:57.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:57.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:57.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:57.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:57.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:04:57.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:04:57.968 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:04:57.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:04:57.968814+0000) 2019-10-18T05:04:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:58.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:58.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:04:59.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:59.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:04:59.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:04:59.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:00.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:00.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:00.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:00.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:01.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:01.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:01.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:01.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:01.959 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:05:01.954+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:01.959640+0000) 2019-10-18T05:05:02.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:02.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:02.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:02.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:02.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:02.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:03.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:03.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:03.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:03.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:03.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:05:03.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:03.764979+0000) 2019-10-18T05:05:03.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:05:03.800+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:03.801620+0000) 2019-10-18T05:05:04.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:04.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:04.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:04.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:05.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:05.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:05.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:05.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:06.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:06.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:06.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:06.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:07.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:07.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:07.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:07.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:07.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:07.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:07.968 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:05:07.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:07.969000+0000) 2019-10-18T05:05:08.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:08.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:08.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:08.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:09.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:09.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:09.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:09.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:10.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:10.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:10.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:10.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:11.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:11.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:11.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:11.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:11.959 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:05:11.954+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:11.959815+0000) 2019-10-18T05:05:12.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:12.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:12.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:12.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:12.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:12.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:13.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:13.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:13.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:13.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:13.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:05:13.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:13.765083+0000) 2019-10-18T05:05:13.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:05:13.800+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:13.801723+0000) 2019-10-18T05:05:14.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:14.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:14.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:14.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:15.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:15.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:16.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:16.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:16.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:16.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:17.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:17.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:17.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:17.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:17.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:17.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:17.968 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:05:17.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:17.969203+0000) 2019-10-18T05:05:18.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:18.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:18.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:18.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:19.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:19.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:19.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:19.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:20.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:20.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:20.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:20.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:21.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:21.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:21.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:21.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:21.959 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:05:21.954+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:21.960016+0000) 2019-10-18T05:05:22.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:22.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:22.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:22.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:22.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:22.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:23.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:23.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:23.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:23.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:23.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:05:23.760+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:23.765186+0000) 2019-10-18T05:05:23.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:05:23.800+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:23.801820+0000) 2019-10-18T05:05:24.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:24.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:24.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:24.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:25.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:25.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:25.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:25.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:26.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:26.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:26.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:26.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:27.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:27.860+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:27.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:27.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:27.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:27.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:27.969 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:05:27.962+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:27.969403+0000) 2019-10-18T05:05:28.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:28.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:28.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:28.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:29.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:29.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:29.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:29.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:30.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:30.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:30.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:30.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:31.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:31.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:31.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:31.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:31.960 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:05:31.958+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:31.960169+0000) 2019-10-18T05:05:32.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:32.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:32.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:32.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:32.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:32.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:33.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:05:33.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:33.765280+0000) 2019-10-18T05:05:33.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:05:33.799+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:33.801918+0000) 2019-10-18T05:05:33.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:33.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:33.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:33.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:34.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:34.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:34.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:34.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:36.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:36.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:36.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:36.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:37.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:37.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:37.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:37.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:37.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:37.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:37.969 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:05:37.966+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:37.969569+0000) 2019-10-18T05:05:38.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:38.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:38.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:38.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:39.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:39.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:39.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:39.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:40.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:40.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:40.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:40.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:41.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:41.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:41.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:41.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:41.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:05:41.958+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:41.960395+0000) 2019-10-18T05:05:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:42.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:42.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:42.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:42.868 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:42.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:43.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:43.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:43.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:43.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:43.773 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:05:43.759+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:43.765382+0000) 2019-10-18T05:05:43.805 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:05:43.799+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:43.802018+0000) 2019-10-18T05:05:44.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:44.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:44.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:44.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:45.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:45.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:45.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:45.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:46.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:46.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:46.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:46.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:47.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:47.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:47.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:47.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:47.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:47.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:47.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:05:47.965+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:47.969748+0000) 2019-10-18T05:05:48.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:48.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:48.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:48.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:49.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:49.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:49.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:49.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:50.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:50.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:50.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:50.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:51.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:51.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:51.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:51.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:51.960 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:05:51.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:51.960578+0000) 2019-10-18T05:05:52.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:52.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:52.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:52.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:52.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:52.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:53.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:53.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:53.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:53.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:53.765 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:05:53.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:53.765464+0000) 2019-10-18T05:05:53.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:05:53.799+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:53.802112+0000) 2019-10-18T05:05:54.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:54.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:54.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:54.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:55.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:55.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:55.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:55.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:56.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:56.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:56.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:56.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:57.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:57.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:57.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:57.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:05:57.863+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:05:57.969 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:05:57.965+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:05:57.969929+0000) 2019-10-18T05:05:58.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:58.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:58.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:58.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:05:59.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:59.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:05:59.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:05:59.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:00.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:00.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:00.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:00.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:01.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:01.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:01.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:01.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:01.960 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:06:01.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:01.960762+0000) 2019-10-18T05:06:02.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:02.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:02.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:02.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:02.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:02.867+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:03.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:03.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:03.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:03.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:03.766 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:03.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:03.765553+0000) 2019-10-18T05:06:03.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:03.762+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:03.766996+0000) 2019-10-18T05:06:03.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:06:03.798+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:03.802211+0000) 2019-10-18T05:06:04.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:04.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:04.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:04.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:05.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:05.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:06.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:06.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:06.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:06.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:07.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:07.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:07.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:07.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:07.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:07.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:07.969 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:06:07.965+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:07.970150+0000) 2019-10-18T05:06:08.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:08.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:08.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:08.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:08.266 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:08.262+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:08.265751+0000) 2019-10-18T05:06:09.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:09.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:09.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:09.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:10.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:10.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:10.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:10.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:11.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:11.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:11.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:11.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:11.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:06:11.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:11.960928+0000) 2019-10-18T05:06:12.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:12.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:12.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:12.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:12.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:12.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:13.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:13.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:13.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:13.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:13.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:06:13.798+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:13.802306+0000) 2019-10-18T05:06:14.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:14.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:14.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:14.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:15.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:15.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:15.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:15.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:16.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:16.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:16.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:16.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:17.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:17.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:17.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:17.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:17.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:17.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:17.970 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:06:17.969+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:17.970405+0000) 2019-10-18T05:06:18.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:18.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:18.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:18.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:18.266 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:18.262+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:18.265852+0000) 2019-10-18T05:06:19.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:19.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:19.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:19.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:20.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:20.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:20.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:20.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:21.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:21.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:21.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:21.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:21.960 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:06:21.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:21.961101+0000) 2019-10-18T05:06:22.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:22.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:22.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:22.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:22.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:22.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:23.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:23.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:23.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:23.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:23.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:06:23.798+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:23.802403+0000) 2019-10-18T05:06:24.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:24.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:24.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:24.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:25.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:25.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:25.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:26.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:26.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:26.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:26.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:27.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:27.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:27.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:27.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:27.875 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:27.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:27.970 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:06:27.969+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:27.970603+0000) 2019-10-18T05:06:28.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:28.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:28.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:28.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:28.266 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:28.262+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:28.265962+0000) 2019-10-18T05:06:29.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:29.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:29.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:29.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:30.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:30.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:30.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:30.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:31.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:31.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:31.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:31.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:31.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:06:31.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:31.961289+0000) 2019-10-18T05:06:32.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:32.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:32.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:32.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:32.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:32.870+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:33.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:33.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:33.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:33.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:33.802 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:06:33.798+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:33.802504+0000) 2019-10-18T05:06:34.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:34.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:34.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:34.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:35.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:35.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:35.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:35.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:36.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:36.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:36.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:36.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:37.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:37.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:37.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:37.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:37.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:37.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:37.970 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:06:37.969+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:37.970815+0000) 2019-10-18T05:06:38.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:38.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:38.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:38.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:38.267 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:38.261+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:38.266069+0000) 2019-10-18T05:06:39.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:39.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:39.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:39.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:40.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:40.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:40.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:40.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:41.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:41.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:41.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:41.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:41.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:06:41.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:41.961484+0000) 2019-10-18T05:06:42.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:42.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:42.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:42.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:42.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:42.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:43.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:43.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:43.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:43.803 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:06:43.797+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:43.802604+0000) 2019-10-18T05:06:44.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:44.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:44.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:44.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:45.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:45.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:45.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:45.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:46.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:46.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:46.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:46.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:47.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:47.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:47.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:47.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:47.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:47.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:47.970 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:06:47.969+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:47.971006+0000) 2019-10-18T05:06:48.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:48.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:48.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:48.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:48.266 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:48.261+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:48.266165+0000) 2019-10-18T05:06:49.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:49.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:49.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:49.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:50.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:50.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:50.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:50.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:51.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:51.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:51.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:51.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:51.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:06:51.957+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:51.961648+0000) 2019-10-18T05:06:52.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:52.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:52.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:52.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:52.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:52.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:53.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:53.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:53.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:53.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:53.803 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:06:53.797+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:53.802697+0000) 2019-10-18T05:06:54.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:54.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:54.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:54.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:55.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:55.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:55.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:55.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:56.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:56.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:56.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:56.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:57.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:57.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:57.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:57.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:57.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:06:57.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:06:57.972 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:06:57.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:57.971182+0000) 2019-10-18T05:06:57.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:57.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:57.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:57.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:58.267 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:58.265+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:58.266267+0000) 2019-10-18T05:06:58.267 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:06:58.265+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:06:58.267849+0000) 2019-10-18T05:06:58.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:58.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:58.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:58.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:06:59.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:59.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:06:59.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:06:59.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:00.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:00.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:00.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:00.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:01.967 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:07:01.956+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:01.961808+0000) 2019-10-18T05:07:01.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:01.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:01.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:01.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:02.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:07:02.765+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:02.766478+0000) 2019-10-18T05:07:02.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:02.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:02.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:02.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:02.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:02.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:03.804 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:03.801+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:03.802795+0000) 2019-10-18T05:07:03.804 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:03.801+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:03.803586+0000) 2019-10-18T05:07:03.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:03.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:03.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:03.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:04.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:04.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:04.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:04.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:05.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:05.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:05.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:05.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:06.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:06.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:06.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:06.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:07.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:07.877+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:07.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:07.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:07.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:07.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:07.971 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:07:07.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:07.971359+0000) 2019-10-18T05:07:08.303 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:08.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:08.303001+0000) 2019-10-18T05:07:08.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:08.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:08.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:08.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:10.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:10.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:10.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:10.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:11.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:11.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:11.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:11.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:11.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:07:11.956+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:11.961992+0000) 2019-10-18T05:07:12.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:12.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:12.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:12.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:12.766 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:07:12.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:12.766580+0000) 2019-10-18T05:07:12.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:12.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:12.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:12.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:12.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:12.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:13.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:13.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:13.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:13.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:14.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:14.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:14.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:14.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:15.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:15.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:15.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:15.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:16.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:16.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:16.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:16.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:17.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:17.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:17.971 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:07:17.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:17.971547+0000) 2019-10-18T05:07:18.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:18.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:18.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:18.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:18.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:18.300+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:18.303103+0000) 2019-10-18T05:07:18.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:18.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:18.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:18.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:19.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:19.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:19.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:19.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:20.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:20.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:20.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:20.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:21.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:21.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:21.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:21.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:21.961 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:07:21.960+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:21.962200+0000) 2019-10-18T05:07:22.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:07:22.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:22.766676+0000) 2019-10-18T05:07:22.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:22.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:22.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:22.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:22.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:22.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:24.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:24.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:24.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:24.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:25.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:25.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:25.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:25.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:25.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:25.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:25.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:25.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:26.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:26.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:26.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:26.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:27.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:27.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:27.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:27.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:27.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:27.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:27.971 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:07:27.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:27.971739+0000) 2019-10-18T05:07:28.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:28.300+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:28.303210+0000) 2019-10-18T05:07:28.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:28.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:28.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:28.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:29.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:29.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:29.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:29.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:31.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:31.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:31.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:31.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:31.964 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:07:31.960+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:31.962394+0000) 2019-10-18T05:07:31.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:31.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:31.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:31.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:32.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:07:32.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:32.766780+0000) 2019-10-18T05:07:32.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:32.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:32.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:32.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:32.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:32.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:34.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:34.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:34.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:34.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:34.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:34.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:34.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:34.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:36.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:36.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:36.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:36.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:36.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:36.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:36.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:36.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:37.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:37.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:37.971 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:07:37.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:37.971944+0000) 2019-10-18T05:07:38.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:38.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:38.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:38.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:38.303 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:38.300+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:38.303319+0000) 2019-10-18T05:07:38.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:38.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T04:57:06.366571+0000 (oldest deadline 2019-10-18T04:57:26.366571+0000) 2019-10-18T05:07:38.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:38.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:40.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:40.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:41.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:41.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:41.962 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:07:41.960+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:41.962576+0000) 2019-10-18T05:07:42.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:42.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:42.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:07:42.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:42.766888+0000) 2019-10-18T05:07:42.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:42.884+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:43.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:43.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:44.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:44.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:44.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:44.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:45.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:45.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:47.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:46.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:47.889 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:47.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:47.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:47.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:47.971 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:07:47.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:47.972091+0000) 2019-10-18T05:07:48.303 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:48.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:48.303416+0000) 2019-10-18T05:07:48.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:48.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:49.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:49.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:50.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:50.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:51.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:51.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:51.962 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:07:51.960+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:51.962754+0000) 2019-10-18T05:07:52.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:07:52.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:52.766987+0000) 2019-10-18T05:07:52.889 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:52.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:52.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:52.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:53.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:53.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:54.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:54.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:55.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:55.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:56.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:56.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:57.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:57.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:57.889 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:07:57.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:07:57.972 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:07:57.968+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:57.972265+0000) 2019-10-18T05:07:58.303 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:07:58.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:07:58.303521+0000) 2019-10-18T05:07:58.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:58.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:07:59.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:07:59.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:00.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:00.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:01.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:01.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:01.962 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:01.959+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:01.962932+0000) 2019-10-18T05:08:02.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:08:02.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:02.767086+0000) 2019-10-18T05:08:02.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:02.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:02.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:02.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:03.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:03.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:04.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:04.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:04.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:04.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:05.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:05.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:05.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:05.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:06.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:06.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:06.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:06.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:07.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:07.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:07.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:07.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:07.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:07.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:07.972 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:08:07.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:07.972403+0000) 2019-10-18T05:08:08.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:08:08.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:08.303625+0000) 2019-10-18T05:08:08.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:08.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:08.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:08.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:09.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:09.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:09.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:09.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:10.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:10.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:10.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:10.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:11.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:11.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:11.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:11.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:11.962 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:11.959+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:11.963117+0000) 2019-10-18T05:08:12.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:08:12.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:12.767192+0000) 2019-10-18T05:08:12.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:12.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:12.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:12.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:12.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:12.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:13.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:13.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:13.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:13.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:15.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:14.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:15.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:14.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:15.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:15.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:15.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:15.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:17.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:17.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:17.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:17.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:17.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:17.887+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:17.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:08:17.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:17.972579+0000) 2019-10-18T05:08:18.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:18.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:18.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:18.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:18.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:08:18.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:18.303754+0000) 2019-10-18T05:08:19.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:19.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:19.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:19.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:20.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:20.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:20.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:20.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:21.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:21.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:21.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:21.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:21.963 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:21.959+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:21.963301+0000) 2019-10-18T05:08:22.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:22.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:22.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:22.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:22.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:08:22.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:22.767299+0000) 2019-10-18T05:08:22.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:22.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:23.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:23.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:23.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:23.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:24.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:24.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:25.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:25.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:25.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:25.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:26.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:26.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:26.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:26.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:27.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:27.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:27.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:27.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:27.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:27.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:27.972 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:08:27.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:27.972723+0000) 2019-10-18T05:08:28.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:28.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:28.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:28.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:28.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:08:28.298+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:28.303862+0000) 2019-10-18T05:08:29.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:29.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:29.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:29.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:30.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:30.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:30.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:30.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:31.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:31.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:31.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:31.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:31.963 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:31.959+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:31.963476+0000) 2019-10-18T05:08:32.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:32.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:32.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:32.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:32.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:08:32.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:32.767408+0000) 2019-10-18T05:08:32.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:32.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:33.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:33.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:33.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:33.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:34.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:34.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:34.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:34.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:35.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:35.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:35.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:35.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:36.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:36.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:36.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:36.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:37.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:37.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:37.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:37.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:37.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:37.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:37.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:08:37.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:37.972888+0000) 2019-10-18T05:08:38.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:38.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:38.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:38.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:38.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:08:38.298+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:38.303972+0000) 2019-10-18T05:08:38.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:38.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:38.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:38.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:39.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:39.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:39.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:39.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:41.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:40.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:41.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:40.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:41.963 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:41.959+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:41.963651+0000) 2019-10-18T05:08:41.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:41.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:41.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:41.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:42.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:08:42.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:42.767518+0000) 2019-10-18T05:08:42.898 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:42.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:42.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:42.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:42.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:42.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:43.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:43.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:43.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:43.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:44.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:44.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:44.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:44.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:45.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:45.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:45.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:45.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:46.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:46.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:46.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:46.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:47.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:47.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:47.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:47.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:47.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:47.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:47.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:08:47.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:47.973055+0000) 2019-10-18T05:08:48.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:08:48.298+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:48.304076+0000) 2019-10-18T05:08:48.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:48.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:48.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:48.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:49.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:49.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:49.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:49.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:50.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:50.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:50.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:50.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:51.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:51.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:51.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:51.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:51.963 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:51.959+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:51.963822+0000) 2019-10-18T05:08:51.964 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:51.959+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:51.964946+0000) 2019-10-18T05:08:52.767 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:08:52.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:52.767625+0000) 2019-10-18T05:08:52.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:52.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:52.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:52.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:52.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:52.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:53.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:53.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:53.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:53.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:54.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:54.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:54.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:54.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:55.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:55.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:55.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:55.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:56.465 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:08:56.459+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:56.464223+0000) 2019-10-18T05:08:56.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:56.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:56.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:56.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:57.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:08:57.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:08:57.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:57.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:57.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:57.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:57.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:08:57.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:57.973230+0000) 2019-10-18T05:08:58.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:08:58.297+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:08:58.304182+0000) 2019-10-18T05:08:58.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:58.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:58.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:58.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:08:59.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:59.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:08:59.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:08:59.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:00.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:00.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:00.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:00.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:01.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:01.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:01.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:01.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:02.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:09:02.761+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:02.767733+0000) 2019-10-18T05:09:02.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:02.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:02.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:02.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:02.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:02.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:03.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:03.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:03.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:04.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:04.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:05.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:04.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:06.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:06.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:06.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:06.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:06.464 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:09:06.459+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:06.464406+0000) 2019-10-18T05:09:07.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:07.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:07.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:07.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:07.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:07.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:07.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:09:07.967+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:07.973397+0000) 2019-10-18T05:09:08.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:08.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:08.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:08.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:08.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:09:08.297+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:08.304287+0000) 2019-10-18T05:09:08.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:08.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:08.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:08.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:09.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:09.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:09.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:09.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:10.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:10.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:10.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:10.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:11.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:11.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:11.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:11.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:12.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:09:12.761+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:12.767843+0000) 2019-10-18T05:09:12.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:12.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:12.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:12.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:12.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:12.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:13.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:13.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:13.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:13.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:14.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:14.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:14.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:14.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:15.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:15.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:15.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:15.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:16.464 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:09:16.458+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:16.464588+0000) 2019-10-18T05:09:16.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:16.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:16.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:16.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:17.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:17.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:17.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:17.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:17.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:17.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:17.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:09:17.966+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:17.973559+0000) 2019-10-18T05:09:18.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:09:18.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:18.304397+0000) 2019-10-18T05:09:18.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:18.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:18.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:18.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:19.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:19.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:19.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:19.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:20.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:20.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:20.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:20.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:21.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:21.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:21.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:21.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:22.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:09:22.765+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:22.767947+0000) 2019-10-18T05:09:22.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:22.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:22.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:22.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:22.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:22.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:23.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:23.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:23.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:23.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:24.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:24.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:24.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:24.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:25.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:25.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:25.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:25.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:26.464 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:09:26.458+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:26.464769+0000) 2019-10-18T05:09:26.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:26.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:26.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:26.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:27.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:27.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:27.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:27.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:27.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:27.905+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:27.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:09:27.966+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:27.973701+0000) 2019-10-18T05:09:28.304 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:09:28.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:28.304509+0000) 2019-10-18T05:09:28.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:28.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:28.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:28.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:29.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:29.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:29.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:29.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:30.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:30.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:30.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:30.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:31.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:31.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:31.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:31.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:32.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:09:32.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:32.768047+0000) 2019-10-18T05:09:32.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:32.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:32.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:32.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:32.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:32.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:33.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:33.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:33.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:33.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:34.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:34.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:34.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:34.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:35.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:35.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:35.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:35.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:36.464 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:09:36.458+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:36.464959+0000) 2019-10-18T05:09:36.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:36.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:36.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:36.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:37.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:37.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:37.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:37.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:37.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:37.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:37.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:09:37.966+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:37.973928+0000) 2019-10-18T05:09:38.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:09:38.300+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:38.304621+0000) 2019-10-18T05:09:38.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:38.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:38.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:38.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:39.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:39.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:39.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:39.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:40.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:40.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:40.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:40.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:41.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:41.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:41.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:41.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:42.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:42.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:42.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:42.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:42.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:09:42.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:42.768161+0000) 2019-10-18T05:09:42.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:42.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:43.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:43.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:43.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:43.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:44.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:44.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:44.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:44.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:45.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:45.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:45.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:45.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:46.466 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:09:46.458+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:46.465142+0000) 2019-10-18T05:09:46.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:46.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:46.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:46.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:47.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:47.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:47.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:47.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:47.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:47.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:47.973 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:09:47.966+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:47.974147+0000) 2019-10-18T05:09:48.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:09:48.300+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:48.304723+0000) 2019-10-18T05:09:48.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:48.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:48.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:48.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:49.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:49.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:49.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:49.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:50.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:50.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:50.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:50.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:51.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:51.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:51.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:51.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:52.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:09:52.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:52.768270+0000) 2019-10-18T05:09:52.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:52.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:52.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:52.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:52.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:52.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:53.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:53.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:53.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:53.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:54.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:54.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:54.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:54.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:55.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:55.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:55.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:55.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:56.465 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:09:56.458+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:56.465318+0000) 2019-10-18T05:09:56.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:56.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:56.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:56.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:57.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:57.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:57.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:57.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:57.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:09:57.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:09:57.974 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:09:57.970+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:57.974374+0000) 2019-10-18T05:09:58.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:09:58.300+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:09:58.304833+0000) 2019-10-18T05:09:58.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:58.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:58.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:58.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:09:59.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:59.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:09:59.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:09:59.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:00.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:00.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:00.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:00.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:01.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:01.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:01.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:01.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:02.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:02.764+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:02.768381+0000) 2019-10-18T05:10:02.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:02.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:02.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:02.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:02.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:02.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:03.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:03.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:03.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:03.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:04.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:04.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:04.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:04.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:05.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:05.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:05.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:05.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:06.465 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:06.458+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:06.465492+0000) 2019-10-18T05:10:06.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:06.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:06.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:06.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:07.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:07.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:07.974 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:10:07.974+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:07.974562+0000) 2019-10-18T05:10:08.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:08.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:08.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:08.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:08.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:10:08.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:08.304896+0000) 2019-10-18T05:10:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:09.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:09.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:10.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:10.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:10.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:10.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:11.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:11.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:11.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:11.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:12.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:12.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:12.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:12.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:12.768 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:12.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:12.768482+0000) 2019-10-18T05:10:12.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:12.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:13.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:13.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:13.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:13.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:14.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:14.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:14.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:14.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:15.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:15.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:15.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:15.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:16.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:16.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:16.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:16.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:16.465 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:16.462+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:16.465665+0000) 2019-10-18T05:10:17.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:17.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:17.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:17.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:17.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:17.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:17.974 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:10:17.974+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:17.974732+0000) 2019-10-18T05:10:18.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:18.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:18.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:18.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:18.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:10:18.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:18.305007+0000) 2019-10-18T05:10:18.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:18.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:18.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:18.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:19.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:19.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:19.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:19.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:20.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:20.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:20.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:20.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:21.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:21.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:21.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:21.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:22.769 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:22.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:22.768583+0000) 2019-10-18T05:10:22.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:22.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:22.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:22.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:22.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:22.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:23.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:23.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:24.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:23.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:24.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:24.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:24.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:24.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:25.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:25.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:25.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:25.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:26.465 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:26.461+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:26.465830+0000) 2019-10-18T05:10:26.466 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:26.465+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:26.466859+0000) 2019-10-18T05:10:27.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:27.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:27.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:27.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:27.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:27.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:27.974 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:10:27.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:27.974923+0000) 2019-10-18T05:10:28.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:28.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:28.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:28.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:28.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:10:28.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:28.305098+0000) 2019-10-18T05:10:28.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:28.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:28.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:28.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:29.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:29.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:29.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:29.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:30.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:30.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:30.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:30.966 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:30.965+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:30.966333+0000) 2019-10-18T05:10:31.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:31.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:31.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:31.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:32.769 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:32.763+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:32.768687+0000) 2019-10-18T05:10:32.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:32.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:32.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:32.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:32.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:32.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:33.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:33.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:33.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:33.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:34.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:34.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:34.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:34.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:35.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:35.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:35.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:35.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:36.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:36.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:36.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:36.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:37.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:37.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:37.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:37.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:37.919 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:37.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:37.974 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:10:37.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:37.975138+0000) 2019-10-18T05:10:38.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:10:38.299+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:38.305203+0000) 2019-10-18T05:10:38.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:38.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:38.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:38.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:39.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:39.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:39.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:39.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:40.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:40.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:40.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:40.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:40.966 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:40.965+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:40.966522+0000) 2019-10-18T05:10:41.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:41.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:41.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:41.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:42.769 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:42.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:42.768787+0000) 2019-10-18T05:10:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:42.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:42.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:42.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:42.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:43.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:43.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:43.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:43.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:44.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:44.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:44.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:44.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:45.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:45.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:45.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:45.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:46.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:46.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:46.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:46.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:47.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:47.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:47.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:47.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:47.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:47.914+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:47.975 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:10:47.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:47.975306+0000) 2019-10-18T05:10:48.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:10:48.298+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:48.305304+0000) 2019-10-18T05:10:48.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:48.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:48.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:48.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:49.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:49.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:49.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:49.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:50.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:50.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:50.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:50.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:50.966 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:10:50.965+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:50.966707+0000) 2019-10-18T05:10:51.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:51.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:51.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:51.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:52.769 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:52.762+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:52.768884+0000) 2019-10-18T05:10:52.770 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:52.766+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:52.769960+0000) 2019-10-18T05:10:52.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:52.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:52.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:52.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:52.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:52.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:53.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:53.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:53.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:53.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:54.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:54.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:54.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:54.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:55.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:55.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:55.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:55.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:56.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:56.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:56.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:56.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:57.269 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:10:57.262+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:57.269085+0000) 2019-10-18T05:10:57.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:57.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:57.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:57.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:57.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:10:57.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:10:57.975 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:10:57.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:57.975509+0000) 2019-10-18T05:10:58.305 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:10:58.298+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:10:58.305410+0000) 2019-10-18T05:10:58.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:58.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:58.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:58.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:10:59.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:59.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:10:59.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:10:59.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:00.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:00.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:00.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:00.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:00.967 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:11:00.965+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:00.966855+0000) 2019-10-18T05:11:01.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:01.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:01.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:01.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:02.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:02.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:02.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:02.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:02.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:03.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:03.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:03.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:03.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:04.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:04.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:04.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:04.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:05.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:05.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:05.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:05.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:06.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:06.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:06.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:06.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:07.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:11:07.262+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:07.269186+0000) 2019-10-18T05:11:07.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:07.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:07.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:07.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:07.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:07.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:07.975 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:11:07.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:07.975677+0000) 2019-10-18T05:11:08.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:11:08.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:08.305529+0000) 2019-10-18T05:11:08.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:08.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:08.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:08.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:09.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:09.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:09.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:09.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:10.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:10.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:10.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:10.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:10.966 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:11:10.965+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:10.967041+0000) 2019-10-18T05:11:11.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:11.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:11.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:11.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:12.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:12.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:12.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:12.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:12.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:12.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:13.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:13.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:13.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:13.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:14.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:14.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:14.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:14.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:15.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:15.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:15.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:15.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:16.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:16.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:16.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:16.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:17.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:11:17.261+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:17.269287+0000) 2019-10-18T05:11:17.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:17.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:17.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:17.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:17.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:17.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:17.975 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:11:17.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:17.975911+0000) 2019-10-18T05:11:18.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:11:18.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:18.305633+0000) 2019-10-18T05:11:18.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:18.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:18.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:18.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:19.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:19.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:19.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:19.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:20.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:20.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:20.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:20.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:20.966 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:11:20.965+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:20.967222+0000) 2019-10-18T05:11:21.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:21.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:21.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:21.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:22.927 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:22.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:23.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:23.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:23.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:23.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:24.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:24.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:24.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:24.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:25.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:25.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:25.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:25.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:26.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:26.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:26.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:27.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:27.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:27.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:27.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:27.269 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:11:27.261+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:27.269392+0000) 2019-10-18T05:11:27.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:27.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:27.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:11:27.973+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:27.976110+0000) 2019-10-18T05:11:28.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:28.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:28.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:28.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:28.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:11:28.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:28.305732+0000) 2019-10-18T05:11:29.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:29.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:29.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:29.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:30.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:30.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:30.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:30.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:30.967 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:11:30.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:30.967404+0000) 2019-10-18T05:11:31.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:31.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:31.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:31.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:32.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:32.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:32.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:32.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:32.929 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:32.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:33.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:33.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:33.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:33.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:34.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:34.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:34.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:34.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:35.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:35.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:35.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:35.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:36.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:36.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:36.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:36.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:37.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:37.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:37.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:37.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:37.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:11:37.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:37.269552+0000) 2019-10-18T05:11:37.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:37.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:37.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:11:37.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:37.976337+0000) 2019-10-18T05:11:38.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:38.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:38.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:38.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:38.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:11:38.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:38.305836+0000) 2019-10-18T05:11:39.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:39.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:39.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:39.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:40.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:40.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:40.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:40.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:40.967 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:11:40.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:40.967590+0000) 2019-10-18T05:11:41.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:41.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:41.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:41.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:41.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:41.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:41.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:41.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:42.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:42.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:43.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:43.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:43.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:43.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:44.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:44.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:44.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:44.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:45.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:45.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:45.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:45.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:46.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:46.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:46.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:46.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:47.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:47.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:47.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:47.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:47.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:11:47.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:47.269653+0000) 2019-10-18T05:11:47.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:47.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:47.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:11:47.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:47.976477+0000) 2019-10-18T05:11:47.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:47.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:47.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:47.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:48.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:11:48.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:48.305940+0000) 2019-10-18T05:11:49.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:49.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:49.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:49.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:50.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:50.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:50.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:50.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:50.967 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:11:50.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:50.967774+0000) 2019-10-18T05:11:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:51.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:51.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:51.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:51.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:51.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:51.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:51.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:52.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:52.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:52.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:52.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:52.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:52.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:53.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:53.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:53.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:53.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 444 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:55.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:54.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:55.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:54.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:55.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:55.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:55.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:55.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:56.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:56.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:56.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:56.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:57.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:11:57.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:57.269751+0000) 2019-10-18T05:11:57.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:11:57.928+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:11:57.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:57.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:57.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:57.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:57.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:11:57.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:57.976604+0000) 2019-10-18T05:11:58.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:11:58.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:11:58.306047+0000) 2019-10-18T05:11:58.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:58.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:58.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:58.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:11:59.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:59.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:11:59.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:11:59.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:00.967 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:12:00.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:00.967973+0000) 2019-10-18T05:12:00.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:00.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:00.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:00.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:02.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:02.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:02.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:02.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:02.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:02.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:02.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:02.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:02.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:02.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:03.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:03.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:03.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:03.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:04.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:04.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:05.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:05.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:05.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:05.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:06.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:06.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:06.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:06.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:07.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:12:07.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:07.269849+0000) 2019-10-18T05:12:07.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:07.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:07.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:07.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:07.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:07.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:07.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:12:07.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:07.976730+0000) 2019-10-18T05:12:08.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:12:08.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:08.306143+0000) 2019-10-18T05:12:08.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:08.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:08.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:08.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:09.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:09.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:09.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:09.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:10.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:10.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:10.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:10.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:10.968 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:12:10.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:10.968161+0000) 2019-10-18T05:12:11.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:11.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:11.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:11.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:12.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:12.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:12.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:12.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:13.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:13.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:13.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:13.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:14.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:14.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:14.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:14.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:15.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:15.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:15.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:15.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:16.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:16.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:16.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:16.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:17.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:12:17.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:17.269950+0000) 2019-10-18T05:12:17.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:17.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:17.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:17.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:17.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:17.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:17.988 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:12:17.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:17.976857+0000) 2019-10-18T05:12:18.306 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:12:18.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:18.306247+0000) 2019-10-18T05:12:18.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:18.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:18.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:18.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:19.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:19.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:19.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:19.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:20.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:20.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:20.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:20.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:20.968 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:12:20.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:20.968352+0000) 2019-10-18T05:12:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:21.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:21.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:21.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:22.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:22.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:22.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:22.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:22.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:22.932+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:23.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:23.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:23.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:23.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:24.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:24.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:24.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:24.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:25.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:25.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:25.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:25.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:26.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:26.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:26.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:26.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:27.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:12:27.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:27.270055+0000) 2019-10-18T05:12:27.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:27.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:27.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:12:27.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:27.976987+0000) 2019-10-18T05:12:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:27.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:27.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:28.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:12:28.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:28.306350+0000) 2019-10-18T05:12:28.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:28.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:28.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:28.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:29.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:29.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:29.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:29.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:30.968 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:12:30.964+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:30.968535+0000) 2019-10-18T05:12:30.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:30.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:30.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:30.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:31.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:31.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:31.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:31.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:32.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:32.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:33.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:32.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:32.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:34.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:34.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:34.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:34.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:35.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:35.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:35.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:35.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:36.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:36.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:36.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:36.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:37.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:36.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:37.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:36.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:37.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:12:37.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:37.270157+0000) 2019-10-18T05:12:37.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:37.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:37.976 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:12:37.972+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:37.977114+0000) 2019-10-18T05:12:38.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:38.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:38.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:38.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:38.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:12:38.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:38.306458+0000) 2019-10-18T05:12:39.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:39.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:39.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:39.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:40.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:40.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:40.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:40.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:40.968 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:12:40.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:40.968719+0000) 2019-10-18T05:12:41.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:41.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:41.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:41.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:42.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:42.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:42.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:42.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:42.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:42.935+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:43.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:42.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:43.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:42.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:43.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:43.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:43.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:43.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:44.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:44.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:44.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:44.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:45.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:45.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:45.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:45.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:46.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:46.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:46.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:46.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:47.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:12:47.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:47.270257+0000) 2019-10-18T05:12:47.942 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:47.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:47.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:47.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:47.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:47.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:47.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:12:47.971+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:47.977238+0000) 2019-10-18T05:12:48.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:12:48.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:48.306570+0000) 2019-10-18T05:12:48.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:48.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:48.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:48.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:50.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:50.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:50.968 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:12:50.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:50.968911+0000) 2019-10-18T05:12:51.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:51.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:51.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:51.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:52.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:51.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:52.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:51.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:52.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:52.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:52.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:52.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:52.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:52.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:54.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:54.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:54.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:54.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:55.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:55.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:55.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:55.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:56.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:56.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:56.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:56.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:56.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:56.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:56.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:56.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:57.270 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:12:57.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:57.270360+0000) 2019-10-18T05:12:57.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:12:57.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:12:57.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:57.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:57.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:57.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:57.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:12:57.971+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:57.977360+0000) 2019-10-18T05:12:58.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:12:58.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:12:58.306693+0000) 2019-10-18T05:12:58.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:58.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:58.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:58.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:12:59.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:59.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:12:59.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:12:59.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:00.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:13:00.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:00.969103+0000) 2019-10-18T05:13:01.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:01.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:01.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:01.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:02.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:02.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:02.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:02.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:03.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:03.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:03.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:03.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:04.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:04.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:04.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:05.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:05.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:05.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:05.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:06.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:06.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:06.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:06.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:07.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:07.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:07.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:07.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:07.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:13:07.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:07.270457+0000) 2019-10-18T05:13:07.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:07.938+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:07.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:13:07.971+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:07.977485+0000) 2019-10-18T05:13:08.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:08.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:08.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:08.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:08.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:13:08.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:08.306801+0000) 2019-10-18T05:13:09.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:09.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:09.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:09.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:10.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:10.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:10.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:10.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:10.969 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:13:10.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:10.969284+0000) 2019-10-18T05:13:11.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:11.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:11.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:11.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:12.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:12.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:12.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:12.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:12.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:12.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:13.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:13.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:13.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:13.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:14.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:14.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:14.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:14.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:15.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:15.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:15.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:15.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:16.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:16.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:16.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:16.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:17.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:17.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:17.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:17.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:17.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:13:17.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:17.270557+0000) 2019-10-18T05:13:17.947 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:17.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:17.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:13:17.971+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:17.977619+0000) 2019-10-18T05:13:18.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:18.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:18.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:18.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:18.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:13:18.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:18.306909+0000) 2019-10-18T05:13:19.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:19.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:19.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:19.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:20.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:20.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:20.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:20.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:20.969 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:13:20.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:20.969444+0000) 2019-10-18T05:13:21.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:20.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:21.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:20.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:21.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:21.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:21.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:21.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:22.948 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:22.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:23.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:23.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:23.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:23.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:23.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:23.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:23.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:23.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:24.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:24.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:24.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:24.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:25.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:25.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:25.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:25.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:26.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:26.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:26.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:26.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:27.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:13:27.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:27.270656+0000) 2019-10-18T05:13:27.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:27.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:27.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:13:27.971+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:27.977745+0000) 2019-10-18T05:13:28.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:27.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:28.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:27.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:28.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:13:28.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:28.307014+0000) 2019-10-18T05:13:29.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:29.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:29.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:29.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:30.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:30.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:30.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:30.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:30.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:13:30.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:30.969591+0000) 2019-10-18T05:13:31.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:31.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:31.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:31.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:32.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:32.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:32.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:32.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:32.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:32.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:32.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:33.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:32.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:34.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:34.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:34.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:34.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:35.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:34.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:35.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:34.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:35.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:35.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:35.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:35.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:36.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:36.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:36.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:36.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:37.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:13:37.265+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:37.270754+0000) 2019-10-18T05:13:37.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:37.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:37.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:37.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:37.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:37.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:37.979 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:13:37.971+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:37.977874+0000) 2019-10-18T05:13:38.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:13:38.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:38.307125+0000) 2019-10-18T05:13:38.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:38.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:38.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:38.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:39.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:39.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:39.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:39.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:40.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:40.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:40.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:40.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:40.970 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:13:40.963+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:40.969789+0000) 2019-10-18T05:13:41.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:41.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:41.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:41.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:42.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:42.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:42.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:42.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:42.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:42.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:43.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:43.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:43.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:43.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:44.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:44.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:44.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:44.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:45.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:45.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:45.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:45.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:46.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:46.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:46.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:46.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:47.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:13:47.265+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:47.270855+0000) 2019-10-18T05:13:47.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:47.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:47.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:47.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:47.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:47.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:47.977 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:13:47.970+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:47.978002+0000) 2019-10-18T05:13:48.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:13:48.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:48.307230+0000) 2019-10-18T05:13:48.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:48.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:48.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:48.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:49.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:49.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:49.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:49.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:50.969 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:13:50.962+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:50.969984+0000) 2019-10-18T05:13:50.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:50.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:50.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:50.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:52.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:51.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:52.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:51.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:52.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:52.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:53.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:53.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:53.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:53.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:54.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:54.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:54.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:54.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:55.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:55.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:55.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:55.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:56.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:56.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:56.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:56.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:57.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:57.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:57.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:57.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:57.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:13:57.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:57.270959+0000) 2019-10-18T05:13:57.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:13:57.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:13:57.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:13:57.970+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:57.978128+0000) 2019-10-18T05:13:58.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:58.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:58.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:58.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:13:58.307 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:13:58.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:13:58.307343+0000) 2019-10-18T05:13:59.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:59.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:13:59.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:13:59.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:00.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:00.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:00.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:00.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:00.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:14:00.962+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:00.970139+0000) 2019-10-18T05:14:01.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:01.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:01.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:01.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:02.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:02.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:02.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:02.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:02.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:02.953+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:03.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:03.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:03.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:03.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:04.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:04.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:04.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:04.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:05.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:05.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:05.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:05.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:06.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:06.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:06.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:06.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:07.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:14:07.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:07.271058+0000) 2019-10-18T05:14:07.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:07.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:07.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:07.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:07.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:07.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:07.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:14:07.974+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:07.978246+0000) 2019-10-18T05:14:08.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:14:08.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:08.307449+0000) 2019-10-18T05:14:08.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:08.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:08.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:08.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:09.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:09.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:09.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:09.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:10.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:10.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:10.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:10.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:10.970 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:14:10.966+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:10.970350+0000) 2019-10-18T05:14:11.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:11.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:11.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:11.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:12.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:12.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:12.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:12.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:12.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:12.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:13.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:13.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:13.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:13.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:14.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:14.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:14.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:14.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:15.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:15.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:15.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:15.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:16.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:16.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:16.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:16.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:17.271 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:14:17.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:17.271158+0000) 2019-10-18T05:14:17.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:17.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:17.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:17.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:17.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:17.952+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:17.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:14:17.974+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:17.978361+0000) 2019-10-18T05:14:18.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:14:18.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:18.307557+0000) 2019-10-18T05:14:18.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:18.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:18.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:18.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:19.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:19.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:19.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:19.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:20.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:20.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:20.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:20.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:20.970 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:14:20.966+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:20.970538+0000) 2019-10-18T05:14:21.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:21.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:21.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:21.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:22.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:22.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:22.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:22.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:22.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:22.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:23.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:23.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:23.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:23.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:24.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:24.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:24.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:24.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:25.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:25.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:25.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:26.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:26.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:26.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:26.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:27.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:14:27.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:27.271257+0000) 2019-10-18T05:14:27.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:27.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:27.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:27.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:27.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:27.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:27.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:14:27.974+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:27.978465+0000) 2019-10-18T05:14:28.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:14:28.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:28.307662+0000) 2019-10-18T05:14:28.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:28.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:28.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:28.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:29.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:29.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:29.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:29.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:30.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:30.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:30.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:30.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:30.970 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:14:30.966+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:30.970729+0000) 2019-10-18T05:14:31.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:31.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:31.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:31.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:32.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:32.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:32.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:32.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:32.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:32.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:33.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:33.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:33.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:33.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:34.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:34.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:34.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:34.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:35.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:35.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:35.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:35.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:36.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:36.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:36.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:36.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:37.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:14:37.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:37.271357+0000) 2019-10-18T05:14:37.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:37.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:37.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:37.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:37.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:37.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:37.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:14:37.974+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:37.978574+0000) 2019-10-18T05:14:38.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:14:38.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:38.307775+0000) 2019-10-18T05:14:38.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:38.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:38.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:38.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:39.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:39.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:39.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:39.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:40.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:40.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:40.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:40.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:40.970 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:14:40.966+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:40.970919+0000) 2019-10-18T05:14:41.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:41.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:41.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:41.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:42.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:42.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:42.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:42.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:42.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:42.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:43.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:43.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:43.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:43.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:44.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:44.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:44.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:44.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:45.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:45.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:45.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:45.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:46.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:46.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:46.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:46.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:47.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:14:47.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:47.271459+0000) 2019-10-18T05:14:47.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:47.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:47.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:47.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:47.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:47.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:47.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:14:47.978+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:47.978691+0000) 2019-10-18T05:14:48.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:14:48.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:48.307887+0000) 2019-10-18T05:14:48.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:48.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:48.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:48.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:49.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:49.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:49.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:49.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:50.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:50.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:50.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:50.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:50.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:14:50.970+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:50.971102+0000) 2019-10-18T05:14:51.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:51.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:51.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:51.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:52.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:52.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:52.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:52.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:52.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:52.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:53.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:53.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:53.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:53.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:54.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:54.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:54.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:54.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:55.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:55.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:55.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:55.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:56.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:56.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:56.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:56.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:57.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:14:57.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:57.271561+0000) 2019-10-18T05:14:57.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:57.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:57.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:57.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:57.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:14:57.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:14:57.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:14:57.977+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:57.978797+0000) 2019-10-18T05:14:58.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:14:58.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:14:58.307997+0000) 2019-10-18T05:14:58.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:58.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:58.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:58.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:14:59.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:59.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:14:59.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:14:59.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:00.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:00.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:00.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:00.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:00.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:15:00.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:00.971286+0000) 2019-10-18T05:15:01.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:01.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:01.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:01.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:02.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:02.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:02.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:02.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:02.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:02.959+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:03.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:03.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:03.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:03.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:04.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:04.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:04.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:04.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:05.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:05.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:05.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:05.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:06.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:06.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:06.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:06.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:07.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:15:07.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:07.271662+0000) 2019-10-18T05:15:07.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:07.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:07.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:07.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:07.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:07.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:07.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:07.977+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:07.978910+0000) 2019-10-18T05:15:08.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:15:08.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:08.308099+0000) 2019-10-18T05:15:08.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:08.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:08.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:08.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:09.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:09.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:09.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:09.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:10.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:10.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:10.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:10.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:10.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:15:10.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:10.971477+0000) 2019-10-18T05:15:11.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:11.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:11.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:11.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:12.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:12.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:12.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:12.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:12.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:12.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:13.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:13.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:13.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:13.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:14.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:14.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:14.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:14.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:15.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:15.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:15.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:15.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:16.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:16.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:16.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:16.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:17.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:15:17.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:17.271761+0000) 2019-10-18T05:15:17.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:17.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:17.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:17.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:17.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:17.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:17.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:17.977+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:17.979027+0000) 2019-10-18T05:15:18.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:15:18.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:18.308207+0000) 2019-10-18T05:15:18.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:18.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:18.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:18.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:19.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:19.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:19.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:19.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:20.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:20.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:20.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:20.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:20.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:15:20.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:20.971663+0000) 2019-10-18T05:15:21.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:21.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:21.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:21.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:22.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:22.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:22.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:22.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:22.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:22.962+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:23.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:23.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:23.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:23.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:24.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:24.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:24.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:24.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:25.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:25.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:25.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:25.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:26.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:26.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:26.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:26.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:27.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:15:27.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:27.271859+0000) 2019-10-18T05:15:27.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:27.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:27.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:27.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:27.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:27.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:27.978 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:27.977+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:27.979189+0000) 2019-10-18T05:15:27.980 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:27.977+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:27.980310+0000) 2019-10-18T05:15:28.308 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:15:28.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:28.308311+0000) 2019-10-18T05:15:28.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:28.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:28.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:28.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:29.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:29.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:29.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:29.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:30.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:30.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:30.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:30.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:30.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:15:30.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:30.971847+0000) 2019-10-18T05:15:31.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:31.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:31.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:31.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:32.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:32.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:32.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:32.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:32.479 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:32.477+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:32.479647+0000) 2019-10-18T05:15:32.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:32.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:33.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:33.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:33.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:33.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:34.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:34.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:34.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:34.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:35.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:35.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:35.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:35.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:36.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:36.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:36.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:36.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:37.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:15:37.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:37.271967+0000) 2019-10-18T05:15:37.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:37.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:37.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:37.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:37.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:37.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:38.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:15:38.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:38.308422+0000) 2019-10-18T05:15:38.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:38.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:38.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:38.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:39.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:39.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:39.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:39.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:40.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:40.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:40.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:40.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:40.971 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:15:40.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:40.972032+0000) 2019-10-18T05:15:41.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:41.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:41.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:41.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:42.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:42.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:42.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:42.479 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:42.477+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:42.479835+0000) 2019-10-18T05:15:42.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:42.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:43.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:43.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:43.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:43.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:44.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:44.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:44.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:45.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:45.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:45.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:45.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:46.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:46.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:46.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:46.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:47.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:15:47.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:47.272072+0000) 2019-10-18T05:15:47.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:47.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:47.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:47.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:47.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:47.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:48.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:15:48.302+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:48.308523+0000) 2019-10-18T05:15:48.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:48.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:48.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:48.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:49.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:49.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:49.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:49.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:50.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:50.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:50.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:50.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:50.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:15:50.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:50.972219+0000) 2019-10-18T05:15:51.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:51.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:51.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:51.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:52.480 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:15:52.477+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:52.480026+0000) 2019-10-18T05:15:52.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:52.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:52.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:52.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:52.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:52.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:53.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:53.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:53.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:53.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:54.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:54.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:54.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:54.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:55.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:55.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:55.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:55.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:56.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:56.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:56.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:56.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:57.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:15:57.265+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:57.272176+0000) 2019-10-18T05:15:57.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:57.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:57.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:57.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:57.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:15:57.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:15:58.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:15:58.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:15:58.308636+0000) 2019-10-18T05:15:58.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:58.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:58.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:58.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:15:59.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:59.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:15:59.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:15:59.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:00.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:00.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:00.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:00.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:00.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:16:00.969+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:00.972405+0000) 2019-10-18T05:16:01.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:01.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:01.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:01.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:02.480 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:16:02.477+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:02.480208+0000) 2019-10-18T05:16:02.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:02.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:02.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:02.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:02.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:02.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:03.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:03.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:03.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:03.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:04.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:04.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:04.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:04.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:05.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:05.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:05.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:05.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:06.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:06.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:06.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:06.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:07.272 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:16:07.265+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:07.272279+0000) 2019-10-18T05:16:07.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:07.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:07.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:07.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:07.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:07.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:08.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:16:08.301+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:08.308740+0000) 2019-10-18T05:16:08.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:08.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:08.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:08.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:09.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:09.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:09.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:09.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:10.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:10.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:10.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:10.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:10.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:16:10.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:10.972588+0000) 2019-10-18T05:16:11.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:11.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:11.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:11.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:12.480 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:16:12.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:12.480413+0000) 2019-10-18T05:16:12.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:12.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:12.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:12.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:12.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:12.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:13.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:13.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:13.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:13.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:14.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:14.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:14.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:14.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:15.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:15.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:16.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:16.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:16.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:16.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:17.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:16:17.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:17.272384+0000) 2019-10-18T05:16:17.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:17.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:17.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:17.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:17.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:17.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:18.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:16:18.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:18.308837+0000) 2019-10-18T05:16:18.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:18.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:18.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:18.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:19.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:19.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:19.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:19.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:20.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:20.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:20.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:20.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:20.980 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:16:20.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:20.972772+0000) 2019-10-18T05:16:21.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:21.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:21.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:21.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:22.485 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:16:22.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:22.480594+0000) 2019-10-18T05:16:22.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:22.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:22.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:22.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:22.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:22.977+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:23.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:23.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:23.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:23.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:24.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:24.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:24.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:24.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:25.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:25.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:25.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:25.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:26.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:26.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:26.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:26.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:27.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:16:27.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:27.272487+0000) 2019-10-18T05:16:27.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:27.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:27.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:27.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:27.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:27.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:28.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:16:28.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:28.308940+0000) 2019-10-18T05:16:28.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:28.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:28.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:28.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:29.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:29.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:29.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:29.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:30.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:30.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:30.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:30.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:30.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:16:30.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:30.972956+0000) 2019-10-18T05:16:31.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:31.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:31.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:31.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:32.480 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:16:32.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:32.480778+0000) 2019-10-18T05:16:32.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:32.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:32.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:32.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:32.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:32.976+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:33.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:33.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:33.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:33.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:34.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:34.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:34.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:34.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:35.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:35.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:35.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:35.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:36.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:36.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:36.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:36.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:37.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:16:37.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:37.272587+0000) 2019-10-18T05:16:37.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:37.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:37.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:37.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:37.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:37.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:38.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:16:38.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:38.309051+0000) 2019-10-18T05:16:38.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:38.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:38.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:38.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:39.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:39.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:39.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:39.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:40.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:40.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:40.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:40.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:40.973 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:16:40.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:40.973139+0000) 2019-10-18T05:16:41.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:41.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:41.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:41.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:42.480 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:16:42.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:42.480951+0000) 2019-10-18T05:16:42.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:42.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:42.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:42.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:42.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:42.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:43.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:43.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:43.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:43.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:44.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:44.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:44.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:44.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:45.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:45.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:45.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:45.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:46.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:46.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:46.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:46.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:47.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:16:47.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:47.272689+0000) 2019-10-18T05:16:47.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:47.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:47.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:47.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:47.984 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:47.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:48.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:16:48.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:48.309148+0000) 2019-10-18T05:16:48.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:48.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:48.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:48.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:49.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:49.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:49.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:49.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:50.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:50.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:50.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:50.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:50.973 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:16:50.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:50.973324+0000) 2019-10-18T05:16:51.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:51.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:51.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:51.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:52.481 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:16:52.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:52.481137+0000) 2019-10-18T05:16:52.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:52.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:52.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:52.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:52.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:52.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:53.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:53.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:53.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:53.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:54.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:54.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:54.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:54.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:55.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:55.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:55.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:55.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:56.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:56.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:57.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:16:57.268+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:57.272793+0000) 2019-10-18T05:16:57.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:57.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:57.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:57.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:57.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:16:57.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:16:58.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:16:58.304+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:16:58.309246+0000) 2019-10-18T05:16:58.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:58.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:58.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:58.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:16:59.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:59.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:16:59.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:16:59.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:00.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:00.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:00.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:00.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:00.973 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:17:00.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:00.973508+0000) 2019-10-18T05:17:01.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:01.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:01.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:01.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:02.481 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:17:02.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:02.481323+0000) 2019-10-18T05:17:02.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:02.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:02.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:02.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:02.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:02.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:03.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:03.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:03.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:03.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:04.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:04.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:04.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:04.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:05.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:05.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:05.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:05.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:06.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:06.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:06.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:06.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:07.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:17:07.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:07.272891+0000) 2019-10-18T05:17:07.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:07.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:07.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:07.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:07.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:08.309 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:17:08.303+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:08.309344+0000) 2019-10-18T05:17:08.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:08.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:08.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:08.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:09.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:09.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:09.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:09.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:10.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:10.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:10.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:10.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:10.974 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:17:10.968+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:10.973702+0000) 2019-10-18T05:17:11.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:11.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:11.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:11.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:12.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:12.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:12.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:12.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:12.482 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:17:12.476+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:12.481514+0000) 2019-10-18T05:17:12.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:12.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:13.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:13.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:13.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:13.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:14.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:14.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:14.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:14.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:15.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:15.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:15.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:15.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:16.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:16.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:16.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:16.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:17.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:17:17.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:17.272995+0000) 2019-10-18T05:17:17.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:17.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:17.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:17.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:17.989 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:17.983+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:18.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:17:18.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:18.309450+0000) 2019-10-18T05:17:18.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:18.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:18.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:18.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:19.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:19.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:19.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:19.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:20.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:20.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:20.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:20.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:20.973 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:17:20.967+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:20.973883+0000) 2019-10-18T05:17:21.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:21.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:21.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:21.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:22.482 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:17:22.475+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:22.481704+0000) 2019-10-18T05:17:22.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:22.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:22.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:22.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:22.990 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:22.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:23.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:23.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:23.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:23.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:24.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:24.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:24.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:24.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:25.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:25.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:25.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:25.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:26.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:26.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:26.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:26.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:27.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:17:27.267+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:27.273100+0000) 2019-10-18T05:17:27.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:27.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:27.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:27.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:27.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:27.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:28.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:17:28.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:28.309552+0000) 2019-10-18T05:17:28.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:28.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:28.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:28.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:29.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:29.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:29.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:29.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:30.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:30.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:30.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:30.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:30.974 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:17:30.967+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:30.974073+0000) 2019-10-18T05:17:31.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:31.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:31.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:31.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:32.481 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:17:32.475+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:32.481894+0000) 2019-10-18T05:17:32.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:32.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:32.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:32.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:32.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:33.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:33.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:33.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:33.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:34.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:34.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:34.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:34.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:35.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:35.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:35.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:35.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:36.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:36.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:36.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:36.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:37.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:17:37.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:37.273200+0000) 2019-10-18T05:17:37.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:37.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:37.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:37.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:37.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:37.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:38.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:17:38.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:38.309671+0000) 2019-10-18T05:17:38.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:38.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:38.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:38.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:39.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:39.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:39.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:39.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:40.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:40.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:40.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:40.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:40.974 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:17:40.971+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:40.974254+0000) 2019-10-18T05:17:41.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:41.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:41.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:41.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:42.482 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:17:42.475+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:42.482084+0000) 2019-10-18T05:17:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:42.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:42.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:42.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:42.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:42.986+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:43.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:43.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:43.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:43.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:44.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:44.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:44.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:44.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:45.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:45.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:45.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:45.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:46.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:46.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:46.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:46.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:47.273 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:17:47.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:47.273300+0000) 2019-10-18T05:17:47.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:47.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:47.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:47.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:47.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:47.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:48.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:17:48.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:48.309770+0000) 2019-10-18T05:17:48.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:48.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:48.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:48.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:49.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:49.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:49.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:49.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:50.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:50.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:50.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:50.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:50.974 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:17:50.971+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:50.974438+0000) 2019-10-18T05:17:51.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:51.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:51.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:51.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:52.482 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:17:52.479+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:52.482266+0000) 2019-10-18T05:17:52.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:52.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:52.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:52.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:52.995 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:52.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:53.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:53.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:53.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:53.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:54.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:54.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:54.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:54.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:55.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:55.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:55.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:55.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:56.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:56.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:56.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:56.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:57.281 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:17:57.266+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:57.273405+0000) 2019-10-18T05:17:57.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:57.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:57.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:57.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:57.996 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:17:57.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:17:58.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:17:58.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:17:58.309879+0000) 2019-10-18T05:17:58.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:58.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:58.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:58.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:17:59.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:59.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:17:59.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:17:59.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:00.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:00.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:00.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:00.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:00.974 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:18:00.971+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:00.974622+0000) 2019-10-18T05:18:01.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:01.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:01.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:01.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:02.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:18:02.479+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:02.482460+0000) 2019-10-18T05:18:02.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:02.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:02.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:02.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:02.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:02.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:03.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:03.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:03.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:03.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:04.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:04.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:04.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:04.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:05.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:05.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:05.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:05.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:06.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:06.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:06.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:06.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:07.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:18:07.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:07.273505+0000) 2019-10-18T05:18:07.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:07.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:07.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:07.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:07.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:07.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:08.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:18:08.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:08.309978+0000) 2019-10-18T05:18:08.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:08.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:08.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:08.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:09.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:09.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:09.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:09.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:10.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:10.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:10.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:10.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:10.974 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:18:10.971+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:10.974812+0000) 2019-10-18T05:18:11.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:11.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:11.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:11.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:12.482 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:18:12.479+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:12.482650+0000) 2019-10-18T05:18:12.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:12.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:12.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:12.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:12.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:12.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:13.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:13.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:13.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:13.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:14.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:14.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:14.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:14.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:15.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:15.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:15.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:15.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:16.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:16.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:16.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:16.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:17.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:18:17.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:17.273609+0000) 2019-10-18T05:18:17.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:17.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:17.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:17.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:17.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:17.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:18.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:18:18.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:18.310070+0000) 2019-10-18T05:18:18.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:18.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:07:43.997560+0000 (oldest deadline 2019-10-18T05:08:03.997560+0000) 2019-10-18T05:18:18.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:18.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:19.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:19.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:20.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:20.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:20.975 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:18:20.971+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:20.975019+0000) 2019-10-18T05:18:21.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:21.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:22.483 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:18:22.479+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:22.482841+0000) 2019-10-18T05:18:22.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:22.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:23.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:22.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:23.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:23.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:24.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:24.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:25.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:25.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:26.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:26.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:27.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:18:27.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:27.273712+0000) 2019-10-18T05:18:27.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:27.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:28.001 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:27.993+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:28.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:18:28.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:28.310175+0000) 2019-10-18T05:18:28.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:28.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:29.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:29.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:30.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:30.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:30.975 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:18:30.970+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:30.975201+0000) 2019-10-18T05:18:31.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:31.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:32.483 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:18:32.478+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:32.483023+0000) 2019-10-18T05:18:32.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:32.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:33.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:32.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:33.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:33.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:34.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:34.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:35.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:35.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:36.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:36.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:37.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:18:37.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:37.273850+0000) 2019-10-18T05:18:37.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:37.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:38.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:38.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:38.310 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:18:38.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:38.310286+0000) 2019-10-18T05:18:38.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:38.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:39.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:39.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:40.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:40.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:40.975 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:18:40.970+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:40.975392+0000) 2019-10-18T05:18:41.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:41.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:42.483 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:18:42.478+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:42.483213+0000) 2019-10-18T05:18:42.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:42.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:43.003 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:43.001+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:43.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:43.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:44.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:44.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:44.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:44.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:45.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:45.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:45.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:45.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:46.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:46.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:46.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:46.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:47.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:18:47.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:47.273949+0000) 2019-10-18T05:18:47.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:47.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:47.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:47.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:48.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:48.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:48.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:18:48.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:48.310388+0000) 2019-10-18T05:18:48.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:48.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:48.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:48.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:49.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:49.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:49.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:49.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:50.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:50.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:50.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:50.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:50.975 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:18:50.970+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:50.975571+0000) 2019-10-18T05:18:51.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:51.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:51.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:51.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:52.496 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:18:52.478+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:52.483404+0000) 2019-10-18T05:18:52.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:52.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:52.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:52.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:53.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:53.000+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:53.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:53.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:53.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:53.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:54.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:54.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:54.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:54.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:55.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:55.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:55.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:55.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:56.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:56.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:56.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:56.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:57.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:18:57.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:57.274052+0000) 2019-10-18T05:18:57.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:57.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:57.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:57.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:58.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:18:58.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:18:58.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:18:58.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:18:58.310502+0000) 2019-10-18T05:18:58.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:58.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:58.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:58.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:18:59.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:59.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:18:59.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:18:59.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:00.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:00.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:00.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:00.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:00.975 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:00.970+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:00.975750+0000) 2019-10-18T05:19:00.976 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:00.970+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:00.976747+0000) 2019-10-18T05:19:01.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:01.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:01.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:01.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:02.483 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:19:02.478+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:02.483587+0000) 2019-10-18T05:19:02.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:02.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:02.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:02.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:03.007 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:03.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:03.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:03.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:03.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:03.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:04.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:04.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:04.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:04.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:05.476 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:05.470+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:05.476285+0000) 2019-10-18T05:19:05.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:05.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:05.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:05.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:06.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:06.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:06.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:06.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:07.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:19:07.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:07.274152+0000) 2019-10-18T05:19:07.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:07.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:07.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:07.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:08.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:08.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:08.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:19:08.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:08.310602+0000) 2019-10-18T05:19:08.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:08.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:08.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:08.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:09.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:09.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:09.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:09.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:10.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:10.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:10.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:10.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:11.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:11.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:11.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:11.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:12.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:19:12.478+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:12.483731+0000) 2019-10-18T05:19:12.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:12.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:12.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:12.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:13.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:13.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:13.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:13.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:13.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:13.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:14.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:14.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:14.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:14.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:15.476 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:15.470+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:15.476506+0000) 2019-10-18T05:19:15.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:15.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:15.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:15.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:16.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:16.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:16.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:16.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:17.274 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:19:17.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:17.274254+0000) 2019-10-18T05:19:17.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:17.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:17.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:17.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:18.009 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:18.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:18.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:19:18.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:18.310705+0000) 2019-10-18T05:19:18.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:18.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:18.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:18.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:19.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:19.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:19.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:19.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:20.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:20.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:20.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:20.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:21.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:21.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:21.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:21.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:22.483 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:19:22.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:22.483919+0000) 2019-10-18T05:19:22.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:22.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:22.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:22.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:23.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:23.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:23.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:23.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:23.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:23.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:24.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:24.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:24.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:24.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:25.476 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:25.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:25.476697+0000) 2019-10-18T05:19:25.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:25.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:25.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:25.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:26.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:26.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:26.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:26.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:27.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:19:27.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:27.274362+0000) 2019-10-18T05:19:27.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:27.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:27.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:27.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:28.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:28.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:28.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:19:28.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:28.310812+0000) 2019-10-18T05:19:28.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:28.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:28.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:28.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:29.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:29.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:29.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:29.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:30.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:30.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:30.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:30.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:31.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:31.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:31.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:31.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:32.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:19:32.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:32.484106+0000) 2019-10-18T05:19:32.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:32.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:32.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:32.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:33.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:33.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:33.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:33.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:33.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:33.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:34.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:34.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:34.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:34.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:35.477 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:35.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:35.476919+0000) 2019-10-18T05:19:35.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:35.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:35.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:35.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:36.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:36.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:36.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:36.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:37.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:19:37.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:37.274462+0000) 2019-10-18T05:19:37.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:37.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:37.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:37.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:38.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:38.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:38.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:19:38.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:38.310926+0000) 2019-10-18T05:19:38.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:38.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:38.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:38.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:39.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:39.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:39.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:39.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:40.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:40.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:40.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:40.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:41.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:41.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:41.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:41.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:42.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:19:42.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:42.484244+0000) 2019-10-18T05:19:42.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:42.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:42.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:42.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:43.013 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:43.007+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:43.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:43.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:43.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:43.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:44.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:44.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:44.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:44.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:45.477 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:45.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:45.477129+0000) 2019-10-18T05:19:45.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:45.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:45.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:45.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:46.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:46.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:46.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:46.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:47.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:19:47.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:47.274562+0000) 2019-10-18T05:19:47.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:47.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:47.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:47.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:48.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:48.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:48.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:19:48.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:48.311029+0000) 2019-10-18T05:19:48.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:48.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:48.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:48.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:49.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:49.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:49.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:49.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:50.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:50.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:50.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:50.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:51.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:51.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:51.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:51.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:52.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:19:52.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:52.484428+0000) 2019-10-18T05:19:52.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:52.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:52.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:52.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:53.015 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:53.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:53.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:53.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:53.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:53.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:54.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:54.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:54.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:54.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:55.479 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:19:55.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:55.477312+0000) 2019-10-18T05:19:55.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:55.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:55.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:55.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:56.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:56.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:56.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:56.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:57.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:19:57.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:57.274661+0000) 2019-10-18T05:19:57.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:57.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:57.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:57.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:58.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:19:58.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:19:58.311 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:19:58.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:19:58.311140+0000) 2019-10-18T05:19:58.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:58.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:58.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:58.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:19:59.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:59.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:19:59.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:19:59.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:00.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:00.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:01.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:01.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:01.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:01.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:02.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:20:02.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:02.484609+0000) 2019-10-18T05:20:02.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:02.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:02.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:02.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:03.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:03.010+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:03.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:03.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:03.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:03.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:04.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:04.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:04.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:04.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:05.477 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:20:05.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:05.477493+0000) 2019-10-18T05:20:05.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:05.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:05.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:05.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:06.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:06.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:06.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:06.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:07.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:20:07.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:07.274760+0000) 2019-10-18T05:20:07.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:07.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:07.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:07.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:08.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:08.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:08.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:20:08.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:08.311239+0000) 2019-10-18T05:20:08.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:08.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:08.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:08.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:09.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:09.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:09.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:09.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:10.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:10.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:10.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:10.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:11.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:11.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:11.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:11.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:12.484 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:20:12.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:12.484805+0000) 2019-10-18T05:20:12.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:12.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:12.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:12.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:13.019 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:13.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:13.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:13.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:13.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:13.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:14.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:14.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:14.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:14.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:15.477 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:20:15.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:15.477677+0000) 2019-10-18T05:20:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:15.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:15.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:15.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:16.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:16.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:16.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:16.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:17.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:20:17.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:17.274864+0000) 2019-10-18T05:20:17.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:17.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:17.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:17.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:18.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:18.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:18.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:20:18.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:18.311349+0000) 2019-10-18T05:20:18.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:18.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:18.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:18.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:19.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:19.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:19.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:19.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:20.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:20.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:20.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:20.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:21.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:21.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:21.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:21.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:22.485 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:20:22.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:22.484989+0000) 2019-10-18T05:20:22.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:22.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:22.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:22.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:23.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:23.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:23.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:23.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:23.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:23.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:24.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:24.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:24.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:24.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:25.477 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:20:25.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:25.477857+0000) 2019-10-18T05:20:25.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:25.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:25.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:25.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:26.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:26.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:26.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:26.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:27.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:20:27.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:27.274963+0000) 2019-10-18T05:20:27.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:27.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:27.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:27.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:28.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:28.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:28.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:20:28.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:28.311448+0000) 2019-10-18T05:20:28.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:28.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:28.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:28.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:29.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:29.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:29.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:29.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:30.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:30.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:30.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:30.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:31.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:31.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:31.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:31.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:32.485 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:20:32.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:32.485174+0000) 2019-10-18T05:20:32.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:32.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:32.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:32.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:33.022 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:33.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:33.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:33.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:33.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:33.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:34.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:34.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:34.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:34.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:35.478 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:20:35.473+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:35.478037+0000) 2019-10-18T05:20:35.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:35.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:35.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:35.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:36.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:36.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:36.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:36.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:37.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:20:37.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:37.275063+0000) 2019-10-18T05:20:37.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:37.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:37.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:37.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:38.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:38.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:38.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:20:38.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:38.311555+0000) 2019-10-18T05:20:38.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:38.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:38.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:38.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:39.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:39.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:39.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:39.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:40.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:40.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:40.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:40.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:41.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:41.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:41.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:41.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:42.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:20:42.481+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:42.485364+0000) 2019-10-18T05:20:42.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:42.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:42.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:42.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:43.025 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:43.017+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:43.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:43.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:43.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:43.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:44.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:44.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:44.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:44.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:45.478 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:20:45.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:45.478221+0000) 2019-10-18T05:20:45.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:45.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:45.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:45.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:46.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:46.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:46.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:46.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:47.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:20:47.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:47.275163+0000) 2019-10-18T05:20:47.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:47.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:47.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:47.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:48.025 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:48.021+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:48.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:20:48.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:48.311666+0000) 2019-10-18T05:20:48.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:48.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:48.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:48.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:49.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:49.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:49.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:49.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:50.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:50.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:50.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:50.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:51.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:51.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:51.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:51.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:52.487 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:20:52.480+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:52.485552+0000) 2019-10-18T05:20:52.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:52.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:52.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:52.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:53.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:53.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:53.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:53.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:53.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:53.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:54.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:54.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:54.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:54.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:55.478 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:20:55.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:55.478401+0000) 2019-10-18T05:20:55.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:55.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:55.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:55.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:56.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:56.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:56.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:56.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:57.275 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:20:57.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:57.275269+0000) 2019-10-18T05:20:57.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:57.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:57.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:57.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:58.027 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:20:58.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:20:58.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:20:58.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:20:58.311781+0000) 2019-10-18T05:20:58.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:58.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:58.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:58.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:20:59.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:59.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:20:59.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:20:59.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:00.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:00.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:00.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:00.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:01.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:01.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:01.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:01.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:02.485 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:21:02.480+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:02.485712+0000) 2019-10-18T05:21:02.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:02.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:02.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:02.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:03.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:03.025+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:03.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:03.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:03.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:03.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:04.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:04.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:04.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:04.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:05.478 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:21:05.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:05.478608+0000) 2019-10-18T05:21:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:05.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:05.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:05.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:06.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:06.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:06.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:06.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:07.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:21:07.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:07.275378+0000) 2019-10-18T05:21:07.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:07.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:07.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:07.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:08.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:08.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:08.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:21:08.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:08.311886+0000) 2019-10-18T05:21:08.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:08.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:08.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:08.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:09.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:09.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:09.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:09.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:10.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:10.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:10.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:10.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:11.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:11.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:11.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:11.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:12.486 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:21:12.480+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:12.485900+0000) 2019-10-18T05:21:12.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:12.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:12.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:12.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:13.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:13.024+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:13.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:13.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:13.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:13.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:14.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:14.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:14.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:14.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:15.478 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:21:15.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:15.478814+0000) 2019-10-18T05:21:15.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:15.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:15.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:15.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:16.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:16.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:16.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:16.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:17.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:21:17.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:17.275479+0000) 2019-10-18T05:21:17.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:17.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:17.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:17.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:18.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:18.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:18.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:21:18.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:18.311999+0000) 2019-10-18T05:21:18.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:18.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:18.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:18.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:19.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:19.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:19.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:19.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:20.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:20.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:20.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:20.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:21.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:21.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:21.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:21.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:22.486 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:21:22.480+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:22.486098+0000) 2019-10-18T05:21:22.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:22.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:22.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:22.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:23.031 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:23.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:23.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:23.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:23.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:23.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:24.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:24.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:24.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:24.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:25.479 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:21:25.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:25.478993+0000) 2019-10-18T05:21:25.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:25.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:25.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:25.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:26.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:26.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:26.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:26.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:27.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:21:27.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:27.275585+0000) 2019-10-18T05:21:27.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:27.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:27.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:27.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:28.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:28.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:28.312 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:21:28.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:28.312106+0000) 2019-10-18T05:21:28.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:28.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:28.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:28.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:29.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:29.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:29.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:29.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:30.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:30.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:30.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:30.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:31.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:31.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:31.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:31.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:32.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:32.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:32.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:32.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:32.488 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:21:32.484+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:32.486254+0000) 2019-10-18T05:21:33.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:33.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:33.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:33.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:33.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:33.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:34.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:34.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:34.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:34.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:35.479 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:21:35.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:35.479180+0000) 2019-10-18T05:21:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:35.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:35.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:35.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:36.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:36.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:36.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:36.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:37.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:21:37.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:37.275684+0000) 2019-10-18T05:21:37.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:37.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:37.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:37.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:38.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:38.027+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:38.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:21:38.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:38.312238+0000) 2019-10-18T05:21:38.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:38.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:38.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:38.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:39.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:39.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:39.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:39.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:40.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:40.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:40.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:40.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:41.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:41.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:41.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:41.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:42.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:42.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:42.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:42.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:42.486 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:21:42.484+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:42.486403+0000) 2019-10-18T05:21:43.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:43.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:43.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:43.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:43.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:43.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:44.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:44.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:44.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:44.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:45.479 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:21:45.476+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:45.479366+0000) 2019-10-18T05:21:45.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:45.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:45.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:45.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:46.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:46.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:46.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:46.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:47.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:21:47.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:47.275784+0000) 2019-10-18T05:21:47.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:47.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:47.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:47.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:48.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:48.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:48.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:21:48.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:48.312348+0000) 2019-10-18T05:21:48.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:48.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:48.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:48.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:49.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:49.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:49.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:49.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:50.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:50.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:50.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:50.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:51.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:51.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:51.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:51.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:52.486 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:21:52.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:52.486586+0000) 2019-10-18T05:21:52.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:52.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:52.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:52.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:53.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:53.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:53.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:53.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:53.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:53.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:54.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:54.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:54.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:54.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:55.480 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:21:55.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:55.479548+0000) 2019-10-18T05:21:55.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:55.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:55.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:55.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:56.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:56.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:56.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:56.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:57.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:21:57.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:57.275857+0000) 2019-10-18T05:21:57.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:57.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:57.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:57.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:58.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:21:58.031+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:21:58.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:21:58.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:21:58.312463+0000) 2019-10-18T05:21:58.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:58.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:58.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:58.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:21:59.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:59.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:21:59.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:21:59.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:00.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:00.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:00.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:00.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:01.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:01.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:01.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:01.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:02.487 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:22:02.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:02.486795+0000) 2019-10-18T05:22:02.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:02.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:02.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:02.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:03.039 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:03.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:03.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:03.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:03.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:03.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:04.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:04.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:04.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:05.479 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:22:05.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:05.479738+0000) 2019-10-18T05:22:05.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:05.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:05.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:05.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:06.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:06.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:06.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:06.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:07.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:22:07.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:07.275964+0000) 2019-10-18T05:22:07.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:07.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:07.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:07.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:08.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:08.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:08.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:22:08.307+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:08.312569+0000) 2019-10-18T05:22:08.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:08.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:08.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:08.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:09.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:09.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:10.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:10.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:10.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:10.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:11.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:11.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:11.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:11.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:12.487 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:22:12.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:12.486974+0000) 2019-10-18T05:22:12.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:12.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:12.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:12.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:13.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:13.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:13.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:13.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:13.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:13.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:14.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:14.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:14.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:14.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:15.480 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:22:15.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:15.479923+0000) 2019-10-18T05:22:15.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:15.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:15.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:15.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:16.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:16.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:16.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:16.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:17.276 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:22:17.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:17.276067+0000) 2019-10-18T05:22:17.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:17.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:17.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:17.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:18.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:18.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:18.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:22:18.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:18.312697+0000) 2019-10-18T05:22:18.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:18.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:18.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:18.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:19.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:19.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:19.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:20.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:20.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:20.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:20.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:21.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:21.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:21.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:21.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:22.487 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:22:22.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:22.487161+0000) 2019-10-18T05:22:22.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:22.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:22.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:22.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:23.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:23.034+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:23.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:23.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:23.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:23.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:24.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:24.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:24.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:24.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:25.480 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:22:25.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:25.480103+0000) 2019-10-18T05:22:25.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:25.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:25.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:25.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:26.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:26.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:26.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:26.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:27.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:22:27.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:27.276167+0000) 2019-10-18T05:22:27.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:27.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:27.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:27.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:28.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:28.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:28.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:22:28.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:28.312793+0000) 2019-10-18T05:22:28.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:28.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:28.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:28.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:29.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:29.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:29.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:29.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:30.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:30.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:30.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:30.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:31.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:31.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:31.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:31.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:32.487 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:22:32.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:32.487352+0000) 2019-10-18T05:22:32.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:32.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:32.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:32.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:33.043 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:33.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:33.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:33.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:33.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:33.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:34.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:34.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:34.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:34.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:35.480 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:22:35.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:35.480279+0000) 2019-10-18T05:22:35.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:35.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:35.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:35.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:36.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:36.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:36.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:36.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:37.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:22:37.270+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:37.276270+0000) 2019-10-18T05:22:37.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:37.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:37.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:37.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:38.043 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:38.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:38.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:22:38.306+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:38.312896+0000) 2019-10-18T05:22:38.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:38.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:38.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:38.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:39.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:39.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:39.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:39.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:40.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:40.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:40.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:40.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:41.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:41.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:41.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:41.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:42.487 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:22:42.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:42.487504+0000) 2019-10-18T05:22:42.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:42.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:42.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:42.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:43.045 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:43.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:43.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:43.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:43.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:43.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:44.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:44.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:44.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:44.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:45.480 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:22:45.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:45.480452+0000) 2019-10-18T05:22:45.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:45.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:45.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:45.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:46.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:46.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:46.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:46.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:47.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:22:47.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:47.276374+0000) 2019-10-18T05:22:47.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:47.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:47.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:47.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:48.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:48.037+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:48.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:22:48.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:48.313002+0000) 2019-10-18T05:22:48.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:48.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:48.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:48.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:49.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:49.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:49.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:49.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:50.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:50.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:50.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:50.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:51.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:51.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:51.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:51.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:52.488 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:22:52.483+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:52.487714+0000) 2019-10-18T05:22:52.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:52.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:52.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:52.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:53.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:53.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:53.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:53.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:53.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:53.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:54.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:54.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:54.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:54.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:55.480 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:22:55.475+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:55.480657+0000) 2019-10-18T05:22:55.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:55.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:55.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:55.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:56.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:56.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:56.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:56.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:57.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:22:57.269+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:57.276476+0000) 2019-10-18T05:22:57.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:57.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:57.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:57.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:58.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:22:58.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:22:58.313 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:22:58.305+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:22:58.313109+0000) 2019-10-18T05:22:58.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:58.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:58.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:58.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:22:59.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:59.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:22:59.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:22:59.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:00.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:00.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:00.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:00.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:01.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:01.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:01.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:01.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:02.488 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:23:02.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:02.487892+0000) 2019-10-18T05:23:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:02.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:02.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:03.048 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:03.041+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:03.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:03.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:03.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:03.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:04.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:04.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:04.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:04.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:05.481 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:23:05.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:05.480865+0000) 2019-10-18T05:23:05.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:05.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:05.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:05.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:06.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:06.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:06.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:06.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:07.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:23:07.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:07.276579+0000) 2019-10-18T05:23:07.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:07.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:07.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:07.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:08.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:08.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:08.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:23:08.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:08.313211+0000) 2019-10-18T05:23:08.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:08.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:08.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:08.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:09.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:09.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:09.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:09.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:10.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:10.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:10.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:10.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:11.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:11.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:11.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:11.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:12.488 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:23:12.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:12.488108+0000) 2019-10-18T05:23:12.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:12.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:12.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:12.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:13.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:13.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:13.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:13.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:13.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:13.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:14.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:14.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:14.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:14.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:15.481 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:23:15.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:15.481064+0000) 2019-10-18T05:23:15.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:15.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:15.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:15.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:16.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:16.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:16.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:16.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:17.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:23:17.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:17.276676+0000) 2019-10-18T05:23:17.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:17.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:17.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:17.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:18.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:18.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:18.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:23:18.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:18.313323+0000) 2019-10-18T05:23:18.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:18.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:18.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:18.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:19.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:19.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:19.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:19.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:20.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:20.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:20.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:20.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:21.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:21.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:21.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:21.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:22.488 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:23:22.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:22.488293+0000) 2019-10-18T05:23:22.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:22.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:22.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:22.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:23.051 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:23.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:23.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:23.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:23.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:23.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:24.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:24.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:24.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:24.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:25.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:23:25.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:25.481243+0000) 2019-10-18T05:23:25.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:25.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:25.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:25.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:26.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:26.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:26.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:26.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:27.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:23:27.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:27.276777+0000) 2019-10-18T05:23:27.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:27.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:27.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:27.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:28.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:28.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:28.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:23:28.308+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:28.313414+0000) 2019-10-18T05:23:28.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:28.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:28.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:28.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:29.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:29.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:29.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:29.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:30.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:30.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:30.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:30.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:31.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:31.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:31.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:31.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:32.488 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:23:32.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:32.488509+0000) 2019-10-18T05:23:32.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:32.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:32.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:32.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:33.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:33.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:33.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:33.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:33.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:33.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:34.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:34.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:34.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:34.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:35.481 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:23:35.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:35.481428+0000) 2019-10-18T05:23:35.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:35.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:35.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:35.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:36.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:36.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:36.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:36.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:37.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:23:37.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:37.276883+0000) 2019-10-18T05:23:37.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:37.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:37.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:37.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:38.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:38.048+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:38.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:23:38.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:38.313744+0000) 2019-10-18T05:23:38.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:38.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:38.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:38.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:39.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:39.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:39.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:39.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:40.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:40.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:40.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:40.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:41.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:41.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:41.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:41.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:42.491 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:23:42.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:42.488696+0000) 2019-10-18T05:23:42.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:42.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:42.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:42.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:43.054 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:43.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:43.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:43.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:43.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:43.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:44.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:44.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:44.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:44.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:45.481 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:23:45.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:45.481635+0000) 2019-10-18T05:23:45.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:45.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:45.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:45.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:46.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:46.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:46.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:46.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:47.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:23:47.272+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:47.276982+0000) 2019-10-18T05:23:47.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:47.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:47.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:47.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:48.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:48.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:48.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:23:48.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:48.313855+0000) 2019-10-18T05:23:48.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:48.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:48.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:48.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:49.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:49.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:49.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:49.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:50.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:50.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:50.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:50.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:51.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:51.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:51.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:51.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:52.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:23:52.482+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:52.488891+0000) 2019-10-18T05:23:52.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:52.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:52.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:52.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:53.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:53.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:53.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:53.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:53.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:53.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:54.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:54.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:54.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:54.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:55.482 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:23:55.474+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:55.481855+0000) 2019-10-18T05:23:55.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:55.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:55.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:55.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:56.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:56.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:56.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:56.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:57.277 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:23:57.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:57.277084+0000) 2019-10-18T05:23:57.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:57.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:57.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:57.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:58.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:23:58.051+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:23:58.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:23:58.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:23:58.313968+0000) 2019-10-18T05:23:58.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:58.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:58.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:58.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:23:59.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:59.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:23:59.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:23:59.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:00.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:00.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:00.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:00.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:01.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:01.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:01.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:01.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:02.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:24:02.486+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:02.489076+0000) 2019-10-18T05:24:02.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:02.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:02.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:02.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:03.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:03.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:03.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:03.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:03.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:03.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:04.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:04.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:04.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:04.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:05.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:24:05.478+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:05.482044+0000) 2019-10-18T05:24:05.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:05.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:05.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:05.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:06.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:06.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:06.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:06.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:07.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:24:07.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:07.277189+0000) 2019-10-18T05:24:07.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:07.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:07.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:07.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:08.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:08.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:08.314 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:24:08.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:08.314073+0000) 2019-10-18T05:24:08.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:08.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:08.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:08.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:09.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:09.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:09.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:09.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:10.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:10.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:10.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:10.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:11.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:11.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:11.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:11.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:12.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:24:12.485+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:12.489256+0000) 2019-10-18T05:24:12.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:12.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:12.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:12.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:13.060 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:13.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:13.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:13.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:13.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:13.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:14.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:14.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:14.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:14.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:15.482 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:24:15.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:15.482227+0000) 2019-10-18T05:24:15.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:15.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:15.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:15.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:16.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:16.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:16.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:16.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:17.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:24:17.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:17.277295+0000) 2019-10-18T05:24:17.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:17.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:17.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:17.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:18.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:18.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:18.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:24:18.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:18.314185+0000) 2019-10-18T05:24:18.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:18.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:18.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:18.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:19.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:19.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:19.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:19.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:20.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:20.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:20.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:20.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:21.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:21.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:21.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:21.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:22.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:24:22.485+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:22.489448+0000) 2019-10-18T05:24:22.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:22.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:22.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:22.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:23.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:23.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:23.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:23.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:23.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:23.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:24.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:24.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:24.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:24.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:25.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:25.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:25.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:25.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:25.482 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:24:25.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:25.482407+0000) 2019-10-18T05:24:26.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:26.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:26.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:26.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:27.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:24:27.271+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:27.277395+0000) 2019-10-18T05:24:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:27.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:27.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:27.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:28.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:28.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:28.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:24:28.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:28.314289+0000) 2019-10-18T05:24:28.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:28.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:28.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:28.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:29.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:29.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:29.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:29.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:30.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:30.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:30.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:30.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:31.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:31.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:31.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:31.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:32.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:32.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:32.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:32.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:32.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:24:32.485+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:32.489658+0000) 2019-10-18T05:24:33.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:33.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:33.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:33.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:33.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:33.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:34.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:34.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:34.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:34.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:35.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:35.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:35.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:35.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:35.482 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:24:35.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:35.482610+0000) 2019-10-18T05:24:36.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:36.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:36.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:36.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:37.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:24:37.274+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:37.277495+0000) 2019-10-18T05:24:37.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:37.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:37.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:37.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:38.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:38.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:38.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:24:38.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:38.314386+0000) 2019-10-18T05:24:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:38.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:38.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:38.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:39.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:39.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:39.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:39.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:40.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:40.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:40.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:40.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:41.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:41.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:41.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:41.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:42.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:42.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:42.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:42.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:42.489 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:24:42.485+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:42.489843+0000) 2019-10-18T05:24:43.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:43.058+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:43.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:43.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:43.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:43.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:44.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:44.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:44.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:44.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:45.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:45.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:45.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:45.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:45.482 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:24:45.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:45.482821+0000) 2019-10-18T05:24:46.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:46.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:46.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:46.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:47.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:24:47.274+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:47.277594+0000) 2019-10-18T05:24:47.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:47.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:47.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:47.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:48.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:48.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:48.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:24:48.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:48.314497+0000) 2019-10-18T05:24:48.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:48.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:48.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:48.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:49.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:49.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:49.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:49.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:50.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:50.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:50.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:50.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:51.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:51.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:51.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:51.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:52.490 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:24:52.485+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:52.490023+0000) 2019-10-18T05:24:52.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:52.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:52.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:52.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:53.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:53.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:53.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:53.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:53.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:53.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:54.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:54.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:54.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:54.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:55.483 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:24:55.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:55.483036+0000) 2019-10-18T05:24:55.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:55.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:55.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:55.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:56.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:56.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:56.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:56.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:57.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:24:57.274+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:57.277702+0000) 2019-10-18T05:24:57.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:57.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:57.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:57.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:58.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:24:58.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:24:58.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:24:58.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:24:58.314608+0000) 2019-10-18T05:24:58.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:58.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:58.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:58.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:24:59.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:59.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:24:59.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:24:59.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:00.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:00.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:00.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:00.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:01.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:01.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:01.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:01.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:02.490 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:25:02.489+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:02.490224+0000) 2019-10-18T05:25:02.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:02.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:02.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:02.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:03.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:03.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:03.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:03.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:03.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:03.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:04.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:04.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:04.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:04.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:05.486 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:25:05.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:05.483205+0000) 2019-10-18T05:25:05.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:05.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:05.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:05.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:06.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:06.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:06.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:06.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:07.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:25:07.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:07.277810+0000) 2019-10-18T05:25:07.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:07.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:07.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:07.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:08.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:08.061+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:08.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:25:08.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:08.314718+0000) 2019-10-18T05:25:08.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:08.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:08.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:08.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:09.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:09.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:09.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:09.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:10.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:10.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:10.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:10.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:11.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:11.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:11.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:11.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:12.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:12.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:12.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:12.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:12.490 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:25:12.489+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:12.490413+0000) 2019-10-18T05:25:13.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:13.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:13.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:13.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:13.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:13.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:14.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:14.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:14.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:14.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:15.483 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:25:15.481+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:15.483364+0000) 2019-10-18T05:25:15.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:15.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:15.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:15.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:16.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:16.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:16.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:16.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:17.281 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:25:17.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:17.277907+0000) 2019-10-18T05:25:17.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:17.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:17.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:17.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:18.071 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:18.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:18.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:25:18.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:18.314834+0000) 2019-10-18T05:25:18.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:18.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:18.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:18.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:19.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:19.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:19.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:19.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:20.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:20.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:20.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:20.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:21.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:21.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:21.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:21.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:22.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:22.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:22.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:22.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:22.490 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:25:22.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:22.490596+0000) 2019-10-18T05:25:23.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:23.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:23.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:23.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:23.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:23.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:24.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:24.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:24.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:24.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:25.483 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:25:25.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:25.483550+0000) 2019-10-18T05:25:25.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:25.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:25.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:25.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:26.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:26.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:26.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:26.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:27.278 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:25:27.277+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:27.278008+0000) 2019-10-18T05:25:27.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:27.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:27.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:27.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:28.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:28.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:28.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:25:28.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:28.314939+0000) 2019-10-18T05:25:28.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:28.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:28.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:28.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:29.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:29.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:29.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:29.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:30.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:30.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:30.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:30.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:31.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:31.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:31.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:31.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:32.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:32.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:32.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:32.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:32.491 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:25:32.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:32.490806+0000) 2019-10-18T05:25:33.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:33.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:33.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:33.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:34.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:34.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:34.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:34.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:35.483 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:25:35.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:35.483728+0000) 2019-10-18T05:25:35.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:35.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:35.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:35.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:36.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:36.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:36.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:36.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:37.279 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:25:37.277+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:37.278110+0000) 2019-10-18T05:25:37.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:37.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:37.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:37.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:38.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:38.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:38.315 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:25:38.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:38.315056+0000) 2019-10-18T05:25:38.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:38.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:38.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:38.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:39.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:39.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:39.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:39.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:40.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:40.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:40.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:40.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:41.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:41.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:41.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:41.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:42.491 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:25:42.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:42.490995+0000) 2019-10-18T05:25:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:42.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:42.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:42.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:43.075 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:43.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:43.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:43.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:43.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:43.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:44.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:44.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:44.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:44.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:45.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:25:45.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:45.483872+0000) 2019-10-18T05:25:45.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:45.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:45.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:45.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:46.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:46.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:46.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:46.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:47.279 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:25:47.276+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:47.278210+0000) 2019-10-18T05:25:47.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:47.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:47.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:47.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:48.076 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:48.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:48.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:25:48.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:48.315156+0000) 2019-10-18T05:25:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:48.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:48.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:48.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:49.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:49.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:49.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:49.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:50.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:50.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:50.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:50.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:51.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:51.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:51.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:51.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:52.491 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:25:52.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:52.491209+0000) 2019-10-18T05:25:52.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:52.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:52.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:52.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:53.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:53.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:53.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:53.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:53.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:53.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:54.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:54.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:54.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:54.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:55.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:55.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:55.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:55.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:55.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:25:55.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:55.484059+0000) 2019-10-18T05:25:56.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:56.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:56.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:56.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:57.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:25:57.276+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:57.278311+0000) 2019-10-18T05:25:57.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:57.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:57.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:57.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:58.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:25:58.072+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:25:58.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:25:58.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:25:58.315262+0000) 2019-10-18T05:25:58.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:58.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:58.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:58.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:25:59.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:59.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:25:59.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:25:59.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:00.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:00.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:00.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:00.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:01.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:01.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:01.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:01.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:02.491 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:26:02.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:02.491390+0000) 2019-10-18T05:26:02.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:02.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:02.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:02.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:03.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:03.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:03.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:03.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:03.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:03.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:04.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:04.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:04.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:04.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:05.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:26:05.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:05.484241+0000) 2019-10-18T05:26:05.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:05.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:05.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:05.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:06.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:06.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:06.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:06.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:07.279 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:26:07.276+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:07.278417+0000) 2019-10-18T05:26:07.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:07.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:07.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:07.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:08.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:08.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:08.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:26:08.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:08.315367+0000) 2019-10-18T05:26:08.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:08.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:08.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:08.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:09.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:09.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:09.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:09.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:10.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:10.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:10.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:10.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:11.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:11.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:11.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:11.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:12.491 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:26:12.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:12.491573+0000) 2019-10-18T05:26:12.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:12.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:12.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:12.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:13.080 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:13.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:13.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:13.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:13.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:13.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:14.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:14.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:14.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:14.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:15.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:26:15.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:15.484424+0000) 2019-10-18T05:26:15.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:15.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:15.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:16.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:16.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:16.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:16.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:17.282 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:26:17.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:17.278715+0000) 2019-10-18T05:26:17.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:17.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:17.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:17.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:18.081 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:18.075+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:18.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:26:18.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:18.315482+0000) 2019-10-18T05:26:18.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:18.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:18.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:18.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:19.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:19.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:19.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:19.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:20.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:20.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:20.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:20.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:21.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:21.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:21.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:21.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:22.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:22.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:22.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:22.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:22.492 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:26:22.488+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:22.491756+0000) 2019-10-18T05:26:23.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:23.075+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:23.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:23.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:23.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:23.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:24.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:24.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:24.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:24.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:25.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:25.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:25.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:25.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:25.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:26:25.480+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:25.484601+0000) 2019-10-18T05:26:26.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:26.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:26.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:26.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:27.279 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:26:27.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:27.278813+0000) 2019-10-18T05:26:27.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:27.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:27.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:27.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:28.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:28.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:28.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:26:28.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:28.315589+0000) 2019-10-18T05:26:28.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:28.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:28.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:28.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:29.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:29.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:29.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:29.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:30.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:30.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:30.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:30.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:31.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:31.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:31.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:31.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:32.492 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:26:32.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:32.491938+0000) 2019-10-18T05:26:32.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:32.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:32.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:32.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:33.084 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:33.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:33.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:33.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:33.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:33.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:34.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:34.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:34.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:34.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:35.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:35.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:35.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:35.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:35.484 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:26:35.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:35.484785+0000) 2019-10-18T05:26:36.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:36.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:36.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:36.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:37.279 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:26:37.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:37.278912+0000) 2019-10-18T05:26:37.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:37.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:37.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:37.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:38.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:38.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:38.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:26:38.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:38.315705+0000) 2019-10-18T05:26:38.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:38.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:38.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:38.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:39.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:39.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:39.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:39.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:40.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:40.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:40.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:40.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:41.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:41.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:41.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:41.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:42.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:42.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:42.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:42.492 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:26:42.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:42.492130+0000) 2019-10-18T05:26:43.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:43.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:43.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:43.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:43.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:43.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:44.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:44.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:44.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:44.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:45.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:45.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:45.485 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:26:45.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:45.484968+0000) 2019-10-18T05:26:46.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:46.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:46.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:46.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:47.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:47.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:47.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:47.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:47.279 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:26:47.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:47.279014+0000) 2019-10-18T05:26:48.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:48.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:48.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:48.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:48.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:48.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:48.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:26:48.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:48.315810+0000) 2019-10-18T05:26:49.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:49.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:49.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:49.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:50.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:50.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:50.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:50.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:51.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:51.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:51.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:51.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:52.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:52.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:52.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:52.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:52.492 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:26:52.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:52.492310+0000) 2019-10-18T05:26:53.087 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:53.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:53.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:53.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:53.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:53.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:54.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:54.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:54.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:54.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 456 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:55.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:55.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:55.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:55.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:55.485 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:26:55.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:55.485144+0000) 2019-10-18T05:26:56.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:56.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:56.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:56.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:57.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:26:57.274+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:57.279116+0000) 2019-10-18T05:26:57.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:57.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:57.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:57.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:58.088 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:26:58.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:26:58.316 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:26:58.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:26:58.315934+0000) 2019-10-18T05:26:58.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:58.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:58.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:58.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:26:59.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:59.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:26:59.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:26:59.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:00.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:00.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:00.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:00.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:01.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:01.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:01.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:01.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:02.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:02.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:02.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:02.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:02.492 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:27:02.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:02.492494+0000) 2019-10-18T05:27:03.088 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:03.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:03.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:03.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:03.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:03.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:04.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:04.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:04.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:04.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:05.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:05.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:05.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:05.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:05.485 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:27:05.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:05.485329+0000) 2019-10-18T05:27:06.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:06.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:06.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:06.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:07.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:07.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:07.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:07.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:07.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:27:07.274+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:07.279216+0000) 2019-10-18T05:27:08.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:08.082+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:08.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:08.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:08.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:08.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:08.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:27:08.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:08.316041+0000) 2019-10-18T05:27:09.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:09.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:09.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:09.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:10.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:10.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:10.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:10.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:11.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:11.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:11.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:11.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:12.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:12.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:12.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:12.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:12.492 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:27:12.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:12.492681+0000) 2019-10-18T05:27:13.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:13.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:13.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:13.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:13.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:13.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:14.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:14.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:14.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:14.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:15.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:15.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:15.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:15.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:15.488 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:27:15.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:15.485510+0000) 2019-10-18T05:27:16.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:16.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:16.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:16.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:17.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:17.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:17.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:17.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:17.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:27:17.274+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:17.279315+0000) 2019-10-18T05:27:18.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:18.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:18.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:18.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:18.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:18.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:18.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:27:18.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:18.316155+0000) 2019-10-18T05:27:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:19.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:19.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:20.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:20.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:20.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:21.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:21.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:21.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:21.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:22.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:22.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:22.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:22.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:22.493 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:27:22.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:22.492874+0000) 2019-10-18T05:27:23.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:23.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:23.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:23.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:23.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:23.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:24.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:24.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:24.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:24.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:25.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:25.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:25.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:25.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:25.485 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:27:25.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:25.485688+0000) 2019-10-18T05:27:26.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:26.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:26.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:26.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:27.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:27:27.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:27.279416+0000) 2019-10-18T05:27:27.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:27.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:27.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:27.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:28.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:28.085+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:28.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:27:28.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:28.316267+0000) 2019-10-18T05:27:28.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:28.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:28.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:28.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:29.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:29.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:29.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:29.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:30.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:30.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:30.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:30.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:31.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:31.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:31.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:31.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:32.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:32.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:32.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:32.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:32.493 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:27:32.487+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:32.493054+0000) 2019-10-18T05:27:33.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:33.085+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:33.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:33.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:33.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:33.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:34.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:34.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:34.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:34.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:35.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:35.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:35.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:35.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:35.486 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:27:35.479+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:35.485873+0000) 2019-10-18T05:27:36.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:36.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:36.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:36.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:37.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:27:37.273+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:37.279515+0000) 2019-10-18T05:27:37.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:37.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:37.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:37.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:38.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:38.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:38.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:27:38.309+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:38.316382+0000) 2019-10-18T05:27:38.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:38.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:38.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:38.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:39.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:39.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:39.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:39.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:40.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:40.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:40.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:40.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:41.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:41.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:41.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:41.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:42.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:42.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:42.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:42.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:42.493 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:27:42.486+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:42.493236+0000) 2019-10-18T05:27:43.095 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:43.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:43.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:43.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:43.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:43.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:44.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:44.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:44.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:44.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:45.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:45.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:45.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:45.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:45.486 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:27:45.478+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:45.486017+0000) 2019-10-18T05:27:46.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:46.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:46.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:46.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:47.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:27:47.277+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:47.279612+0000) 2019-10-18T05:27:47.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:47.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:47.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:47.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:48.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:48.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:48.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:27:48.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:48.316441+0000) 2019-10-18T05:27:48.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:48.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:48.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:48.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:49.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:49.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:49.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:49.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:50.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:50.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:50.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:50.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:51.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:51.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:51.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:51.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:52.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:52.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:52.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:52.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:52.493 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:27:52.486+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:52.493418+0000) 2019-10-18T05:27:53.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:53.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:53.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:53.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:53.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:53.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:54.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:54.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:54.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:54.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:55.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:55.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:55.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:55.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:55.486 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:27:55.482+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:55.486222+0000) 2019-10-18T05:27:56.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:56.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:56.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:56.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:57.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:57.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:57.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:57.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:57.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:27:57.277+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:57.279711+0000) 2019-10-18T05:27:58.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:27:58.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:27:58.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:58.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:58.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:58.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:27:58.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:27:58.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:27:58.316546+0000) 2019-10-18T05:27:59.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:59.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:27:59.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:27:59.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:00.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:00.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:00.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:00.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:01.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:01.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:01.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:01.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:02.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:02.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:02.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:02.493 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:28:02.486+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:02.493599+0000) 2019-10-18T05:28:03.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:03.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:03.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:03.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:03.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:03.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:04.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:04.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:04.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:04.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:05.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:05.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:05.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:05.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:05.486 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:28:05.482+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:05.486399+0000) 2019-10-18T05:28:06.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:06.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:06.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:06.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:07.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:07.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:07.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:07.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:07.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:07.276+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:07.279812+0000) 2019-10-18T05:28:08.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:08.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:08.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:08.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:08.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:08.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:08.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:28:08.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:08.316656+0000) 2019-10-18T05:28:09.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:09.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:09.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:09.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:10.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:10.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:10.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:10.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:11.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:11.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:11.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:11.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:12.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:12.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:12.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:12.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:12.494 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:28:12.486+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:12.493802+0000) 2019-10-18T05:28:13.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:13.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:13.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:13.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:13.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:13.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:14.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:14.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:14.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:14.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:15.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:15.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:15.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:15.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:15.486 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:28:15.482+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:15.486581+0000) 2019-10-18T05:28:16.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:16.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:16.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:16.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:17.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:17.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:17.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:17.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:17.280 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:17.276+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:17.279910+0000) 2019-10-18T05:28:18.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:18.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:18.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:18.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:18.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:18.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:18.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:28:18.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:18.316773+0000) 2019-10-18T05:28:19.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:19.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:19.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:19.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:20.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:20.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:20.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:20.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:21.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:21.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:21.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:21.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:22.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:22.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:22.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:22.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:22.494 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:28:22.486+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:22.494010+0000) 2019-10-18T05:28:23.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:23.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:23.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:23.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:23.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:23.096+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:24.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:24.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:24.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:24.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:25.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:25.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:25.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:25.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:25.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:28:25.482+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:25.486765+0000) 2019-10-18T05:28:26.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:26.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:26.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:26.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:27.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:27.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:27.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:27.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:27.283 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:27.276+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:27.280007+0000) 2019-10-18T05:28:28.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:28.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:28.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:28.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:28.103 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:28.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:28.317 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:28:28.312+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:28.316882+0000) 2019-10-18T05:28:29.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:29.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:29.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:29.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:30.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:30.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:30.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:30.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:31.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:31.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:31.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:31.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:32.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:32.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:32.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:32.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:32.494 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:28:32.490+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:32.494216+0000) 2019-10-18T05:28:33.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:33.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:33.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:33.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:33.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:33.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:34.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:34.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:34.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:34.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:35.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:35.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:35.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:35.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:35.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:28:35.482+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:35.486942+0000) 2019-10-18T05:28:36.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:36.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:36.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:36.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:37.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:37.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:37.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:37.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:37.281 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:37.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:37.280105+0000) 2019-10-18T05:28:38.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:38.099+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:38.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:38.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:38.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:38.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:38.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:28:38.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:38.316998+0000) 2019-10-18T05:28:39.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:39.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:39.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:39.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:40.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:40.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:40.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:40.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:41.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:41.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:41.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:41.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:42.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:42.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:42.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:42.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:42.494 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:28:42.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:42.494357+0000) 2019-10-18T05:28:43.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:43.099+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:43.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:43.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:43.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:43.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:44.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:44.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:44.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:44.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:45.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:45.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:45.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:45.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:45.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:28:45.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:45.487126+0000) 2019-10-18T05:28:46.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:46.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:46.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:46.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:47.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:47.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:47.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:47.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:47.281 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:47.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:47.280199+0000) 2019-10-18T05:28:48.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:48.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:48.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:48.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:48.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:48.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:48.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:28:48.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:48.317100+0000) 2019-10-18T05:28:49.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:49.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:49.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:49.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:50.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:50.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:50.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:50.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:51.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:51.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:51.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:51.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:52.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:52.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:52.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:52.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:52.494 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:28:52.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:52.494531+0000) 2019-10-18T05:28:53.107 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:53.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:53.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:53.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:53.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:53.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:54.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:54.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:55.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:55.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:18:24.227360+0000 (oldest deadline 2019-10-18T05:18:44.227360+0000) 2019-10-18T05:28:55.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:55.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:55.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:28:55.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:55.487306+0000) 2019-10-18T05:28:56.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:56.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:57.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:57.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:57.281 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:57.275+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:57.280288+0000) 2019-10-18T05:28:57.282 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:28:57.279+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:57.281801+0000) 2019-10-18T05:28:58.108 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:28:58.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:28:58.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:58.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:28:58.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:28:58.311+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:28:58.317211+0000) 2019-10-18T05:28:59.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:28:59.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:00.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:00.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:01.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:01.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:01.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:29:01.775+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:01.780479+0000) 2019-10-18T05:29:02.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:02.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:02.495 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:29:02.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:02.494715+0000) 2019-10-18T05:29:03.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:03.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:03.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:03.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:04.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:04.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:05.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:05.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:05.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:29:05.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:05.487483+0000) 2019-10-18T05:29:06.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:06.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:07.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:07.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:08.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:08.103+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:08.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:08.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:08.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:29:08.310+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:08.317324+0000) 2019-10-18T05:29:09.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:09.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:10.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:10.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:11.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:11.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:11.781 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:29:11.774+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:11.780582+0000) 2019-10-18T05:29:12.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:12.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:12.495 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:29:12.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:12.494905+0000) 2019-10-18T05:29:13.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:13.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:13.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:13.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:14.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:14.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:15.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:15.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:15.487 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:29:15.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:15.487663+0000) 2019-10-18T05:29:16.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:16.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:17.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:17.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:18.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:18.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:18.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:29:18.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:18.317440+0000) 2019-10-18T05:29:18.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:18.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:19.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:19.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:19.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:19.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:20.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:20.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:20.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:20.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:21.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:21.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:21.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:21.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:21.781 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:29:21.774+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:21.780685+0000) 2019-10-18T05:29:22.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:22.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:22.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:22.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:22.495 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:29:22.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:22.495102+0000) 2019-10-18T05:29:23.112 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:23.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:23.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:23.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:23.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:23.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:24.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:24.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:24.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:24.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:25.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:25.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:25.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:25.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:25.490 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:29:25.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:25.487844+0000) 2019-10-18T05:29:26.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:26.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:26.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:26.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:27.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:27.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:27.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:27.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:28.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:28.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:28.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:29:28.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:28.317571+0000) 2019-10-18T05:29:28.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:28.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:28.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:28.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:29.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:29.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:29.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:29.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:30.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:30.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:30.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:30.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:31.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:31.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:31.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:31.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:31.781 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:29:31.774+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:31.780790+0000) 2019-10-18T05:29:32.495 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:29:32.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:32.495284+0000) 2019-10-18T05:29:32.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:32.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:32.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:32.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:33.114 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:33.106+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:33.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:33.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:33.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:33.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:34.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:34.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:34.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:34.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:35.488 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:29:35.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:35.488027+0000) 2019-10-18T05:29:35.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:35.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:35.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:35.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:36.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:36.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:36.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:36.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:37.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:37.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:37.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:37.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:38.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:38.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:38.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:29:38.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:38.317700+0000) 2019-10-18T05:29:38.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:38.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:38.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:38.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:39.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:39.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:39.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:39.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:40.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:40.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:40.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:40.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:41.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:41.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:41.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:41.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:41.781 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:29:41.774+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:41.780888+0000) 2019-10-18T05:29:42.495 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:29:42.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:42.495464+0000) 2019-10-18T05:29:42.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:42.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:42.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:42.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:43.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:43.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:43.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:43.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:43.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:43.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:44.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:44.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:44.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:44.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:45.488 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:29:45.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:45.488213+0000) 2019-10-18T05:29:45.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:45.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:45.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:45.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:46.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:46.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:46.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:46.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:47.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:47.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:47.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:47.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:48.116 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:48.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:48.318 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:29:48.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:48.317808+0000) 2019-10-18T05:29:48.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:48.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:48.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:48.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:49.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:49.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:49.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:49.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:50.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:50.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:50.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:50.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:51.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:51.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:51.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:51.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:51.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:29:51.773+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:51.780990+0000) 2019-10-18T05:29:52.495 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:29:52.493+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:52.495647+0000) 2019-10-18T05:29:52.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:52.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:52.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:52.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:53.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:53.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:53.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:53.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:53.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:53.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:54.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:54.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:54.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:54.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:55.488 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:29:55.485+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:55.488394+0000) 2019-10-18T05:29:55.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:55.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:55.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:55.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:56.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:56.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:56.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:56.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:57.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:57.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:57.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:57.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:58.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:29:58.109+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:29:58.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:29:58.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:29:58.317963+0000) 2019-10-18T05:29:58.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:58.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:58.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:58.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:29:59.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:59.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:29:59.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:29:59.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:00.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:00.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:00.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:00.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:01.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:01.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:01.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:01.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:01.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:30:01.773+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:01.781089+0000) 2019-10-18T05:30:02.496 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:30:02.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:02.495832+0000) 2019-10-18T05:30:02.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:02.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:02.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:02.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:03.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:03.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:03.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:03.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:03.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:03.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:04.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:04.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:04.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:04.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:05.488 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:30:05.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:05.488571+0000) 2019-10-18T05:30:05.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:05.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:05.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:05.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:06.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:06.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:06.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:06.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:07.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:07.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:07.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:07.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:08.120 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:08.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:08.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:30:08.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:08.318069+0000) 2019-10-18T05:30:08.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:08.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:08.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:08.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:09.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:09.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:09.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:09.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:10.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:10.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:10.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:10.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:11.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:11.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:11.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:11.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:11.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:30:11.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:11.781188+0000) 2019-10-18T05:30:12.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:30:12.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:12.496018+0000) 2019-10-18T05:30:12.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:12.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:12.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:12.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:13.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:13.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:13.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:13.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:13.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:13.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:14.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:14.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:14.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:14.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:15.489 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:30:15.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:15.488752+0000) 2019-10-18T05:30:15.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:15.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:15.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:15.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:16.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:16.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:16.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:16.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:17.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:17.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:17.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:17.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:18.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:18.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:18.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:30:18.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:18.318180+0000) 2019-10-18T05:30:18.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:18.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:18.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:18.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:19.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:19.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:19.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:19.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:20.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:20.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:20.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:20.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:21.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:21.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:21.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:21.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:21.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:30:21.776+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:21.781289+0000) 2019-10-18T05:30:22.496 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:30:22.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:22.496230+0000) 2019-10-18T05:30:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:22.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:22.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:22.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:23.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:23.116+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:23.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:23.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:23.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:23.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:24.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:24.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:24.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:24.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:25.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:25.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:25.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:25.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:25.489 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:30:25.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:25.488964+0000) 2019-10-18T05:30:26.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:26.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:26.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:26.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:27.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:27.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:27.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:27.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:28.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:28.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:28.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:30:28.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:28.318290+0000) 2019-10-18T05:30:28.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:28.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:28.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:28.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:29.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:29.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:29.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:29.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:30.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:30.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:30.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:30.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:31.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:31.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:31.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:31.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:31.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:30:31.776+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:31.781391+0000) 2019-10-18T05:30:32.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:32.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:32.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:32.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:32.497 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:30:32.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:32.496415+0000) 2019-10-18T05:30:33.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:33.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:33.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:33.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:33.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:33.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:34.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:34.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:34.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:34.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:35.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:35.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:35.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:35.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:35.489 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:30:35.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:35.489144+0000) 2019-10-18T05:30:36.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:36.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:36.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:36.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:37.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:37.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:37.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:37.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:38.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:38.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:38.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:30:38.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:38.318405+0000) 2019-10-18T05:30:38.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:38.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:38.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:38.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:39.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:39.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:39.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:39.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:40.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:40.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:40.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:40.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:41.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:41.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:41.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:41.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:41.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:30:41.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:41.781494+0000) 2019-10-18T05:30:42.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:42.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:42.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:42.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:42.496 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:30:42.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:42.496596+0000) 2019-10-18T05:30:43.126 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:43.120+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:43.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:43.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:43.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:43.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:44.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:44.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:44.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:44.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:45.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:45.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:45.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:45.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:45.489 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:30:45.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:45.489326+0000) 2019-10-18T05:30:46.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:46.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:46.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:46.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:47.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:47.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:47.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:47.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:48.127 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:48.124+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:48.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:30:48.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:48.318512+0000) 2019-10-18T05:30:48.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:48.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:48.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:48.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:49.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:49.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:49.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:49.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:50.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:50.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:50.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:50.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:51.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:51.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:51.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:51.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:51.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:30:51.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:51.781595+0000) 2019-10-18T05:30:52.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:52.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:52.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:52.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:52.497 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:30:52.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:52.496764+0000) 2019-10-18T05:30:53.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:53.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:53.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:53.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:53.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:53.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:54.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:54.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:54.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:54.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:55.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:55.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:55.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:55.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:55.489 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:30:55.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:55.489506+0000) 2019-10-18T05:30:56.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:56.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:56.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:56.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:57.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:57.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:57.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:57.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:58.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:30:58.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:30:58.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:30:58.315+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:30:58.318627+0000) 2019-10-18T05:30:58.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:58.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:58.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:58.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:30:59.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:59.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:30:59.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:30:59.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:00.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:00.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:01.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:01.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:01.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:01.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:01.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:31:01.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:01.781694+0000) 2019-10-18T05:31:02.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:02.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:02.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:02.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:02.497 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:31:02.492+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:02.496953+0000) 2019-10-18T05:31:03.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:03.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:03.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:03.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:03.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:03.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:04.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:04.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:04.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:04.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:05.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:05.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:05.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:05.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:05.490 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:31:05.484+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:05.489691+0000) 2019-10-18T05:31:06.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:06.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:06.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:06.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:07.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:07.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:07.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:07.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:08.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:08.123+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:08.319 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:31:08.315+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:08.318738+0000) 2019-10-18T05:31:08.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:08.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:08.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:08.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:09.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:09.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:09.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:09.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:10.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:10.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:10.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:10.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:11.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:11.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:11.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:11.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:11.782 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:31:11.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:11.781796+0000) 2019-10-18T05:31:12.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:12.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:12.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:12.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:12.497 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:31:12.491+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:12.497142+0000) 2019-10-18T05:31:13.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:13.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:13.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:13.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:13.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:13.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:14.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:14.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:14.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:14.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:15.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:15.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:15.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:15.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:15.490 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:31:15.483+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:15.489903+0000) 2019-10-18T05:31:16.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:16.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:16.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:16.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:17.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:17.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:17.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:17.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:18.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:18.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:18.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:31:18.315+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:18.318852+0000) 2019-10-18T05:31:18.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:18.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:18.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:18.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:19.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:19.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:19.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:19.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:20.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:20.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:20.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:20.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:21.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:21.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:21.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:21.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:21.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:31:21.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:21.781896+0000) 2019-10-18T05:31:22.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:22.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:22.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:22.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:22.497 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:31:22.491+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:22.497362+0000) 2019-10-18T05:31:23.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:23.127+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:23.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:23.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:23.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:23.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:24.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:24.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:24.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:24.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:25.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:25.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:25.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:25.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:25.490 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:31:25.483+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:25.490119+0000) 2019-10-18T05:31:26.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:26.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:26.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:26.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:27.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:27.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:27.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:27.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:28.134 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:28.126+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:28.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:28.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:28.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:28.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:28.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:31:28.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:28.318960+0000) 2019-10-18T05:31:29.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:29.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:29.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:29.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:30.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:30.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:30.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:30.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:31.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:31.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:31.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:31.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:31.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:31:31.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:31.781999+0000) 2019-10-18T05:31:32.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:32.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:32.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:32.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:32.497 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:31:32.491+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:32.497552+0000) 2019-10-18T05:31:33.134 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:33.126+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:33.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:33.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:33.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:33.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:34.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:34.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:34.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:34.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:35.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:35.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:35.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:35.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:35.491 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:31:35.487+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:35.490345+0000) 2019-10-18T05:31:36.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:36.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:36.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:36.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:37.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:37.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:37.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:37.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:38.135 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:38.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:38.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:38.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:38.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:38.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:38.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:31:38.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:38.319089+0000) 2019-10-18T05:31:39.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:39.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:39.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:39.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:40.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:40.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:40.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:40.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:41.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:41.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:41.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:41.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:41.784 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:31:41.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:41.782099+0000) 2019-10-18T05:31:42.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:42.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:42.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:42.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:42.498 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:31:42.491+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:42.497762+0000) 2019-10-18T05:31:43.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:43.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:43.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:43.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:43.135 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:43.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:44.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:44.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:44.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:44.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:45.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:45.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:45.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:45.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:45.490 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:31:45.487+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:45.490524+0000) 2019-10-18T05:31:46.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:46.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:46.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:46.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:47.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:47.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:47.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:47.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:48.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:48.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:48.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:48.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:48.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:48.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:48.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:31:48.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:48.319201+0000) 2019-10-18T05:31:49.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:49.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:49.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:49.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:50.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:50.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:50.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:50.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:51.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:51.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:51.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:51.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:51.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:31:51.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:51.782197+0000) 2019-10-18T05:31:52.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:52.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:52.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:52.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:52.498 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:31:52.491+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:52.497931+0000) 2019-10-18T05:31:53.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:53.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:53.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:53.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:53.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:53.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:54.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:54.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:54.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:54.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:55.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:55.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:55.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:55.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:55.491 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:31:55.487+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:55.490731+0000) 2019-10-18T05:31:56.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:56.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:56.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:56.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:57.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:57.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:57.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:57.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:58.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:31:58.130+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:31:58.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:58.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:58.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:58.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:31:58.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:31:58.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:31:58.319318+0000) 2019-10-18T05:31:59.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:59.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:31:59.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:31:59.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:00.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:00.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:00.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:00.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:01.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:01.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:01.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:01.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:01.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:32:01.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:01.782296+0000) 2019-10-18T05:32:02.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:02.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:02.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:02.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:02.498 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:32:02.491+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:02.498100+0000) 2019-10-18T05:32:03.140 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:03.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:03.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:03.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:03.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:04.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:04.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:04.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:04.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:05.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:05.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:05.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:05.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:05.491 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:32:05.487+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:05.490917+0000) 2019-10-18T05:32:06.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:06.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:06.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:06.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:07.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:07.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:07.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:07.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:08.140 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:08.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:08.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:32:08.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:08.319430+0000) 2019-10-18T05:32:08.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:08.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:08.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:08.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:09.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:09.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:09.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:09.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:10.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:10.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:10.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:10.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:11.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:11.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:11.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:11.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:11.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:32:11.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:11.782398+0000) 2019-10-18T05:32:12.498 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:32:12.495+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:12.498349+0000) 2019-10-18T05:32:12.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:12.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:12.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:12.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:13.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:13.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:13.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:13.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:13.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:13.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:14.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:14.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:14.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:14.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:15.491 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:32:15.487+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:15.491131+0000) 2019-10-18T05:32:15.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:15.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:15.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:15.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:16.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:16.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:16.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:16.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:17.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:17.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:17.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:17.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:18.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:18.133+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:18.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:32:18.313+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:18.319545+0000) 2019-10-18T05:32:18.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:18.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:18.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:18.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:19.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:19.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:19.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:19.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:20.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:20.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:20.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:20.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:21.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:21.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:21.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:21.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:21.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:32:21.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:21.782493+0000) 2019-10-18T05:32:22.499 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:32:22.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:22.498527+0000) 2019-10-18T05:32:22.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:22.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:22.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:22.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:23.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:23.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:23.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:23.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:23.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:23.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:24.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:24.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:24.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:24.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:25.491 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:32:25.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:25.491345+0000) 2019-10-18T05:32:25.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:25.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:25.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:25.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:26.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:26.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:26.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:26.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:27.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:27.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:27.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:27.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:28.143 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:28.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:28.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:32:28.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:28.319658+0000) 2019-10-18T05:32:28.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:28.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:28.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:28.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:29.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:29.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:29.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:29.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:30.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:30.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:30.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:30.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:31.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:31.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:31.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:31.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:31.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:32:31.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:31.782593+0000) 2019-10-18T05:32:32.499 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:32:32.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:32.498705+0000) 2019-10-18T05:32:32.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:32.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:32.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:32.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:33.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:33.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:33.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:33.749+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:33.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:33.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:34.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:34.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:34.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:34.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:35.492 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:32:35.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:35.491528+0000) 2019-10-18T05:32:35.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:35.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:35.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:35.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:36.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:36.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:36.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:36.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:37.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:37.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:37.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:37.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:38.146 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:38.140+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:38.320 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:32:38.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:38.319775+0000) 2019-10-18T05:32:38.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:38.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:38.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:38.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:39.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:39.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:39.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:39.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:40.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:40.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:40.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:40.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:41.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:41.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:41.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:41.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:41.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:32:41.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:41.782696+0000) 2019-10-18T05:32:42.499 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:32:42.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:42.498848+0000) 2019-10-18T05:32:42.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:42.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:42.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:42.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:43.147 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:43.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:43.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:43.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:43.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:43.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:44.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:44.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:44.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:44.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:45.492 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:32:45.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:45.491741+0000) 2019-10-18T05:32:45.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:45.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:45.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:45.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:46.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:46.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:46.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:46.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:47.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:47.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:47.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:47.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:48.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:48.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:48.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:32:48.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:48.319894+0000) 2019-10-18T05:32:48.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:48.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:48.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:48.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:49.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:49.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:49.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:49.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:50.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:50.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:50.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:50.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:51.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:51.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:51.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:51.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:51.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:32:51.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:51.782807+0000) 2019-10-18T05:32:52.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:32:52.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:52.499053+0000) 2019-10-18T05:32:52.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:52.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:52.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:52.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:53.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:53.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:53.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:53.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:53.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:53.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:54.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:54.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:54.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:54.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:55.492 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:32:55.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:55.491951+0000) 2019-10-18T05:32:55.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:55.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:55.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:55.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:56.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:56.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:56.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:56.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:57.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:57.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:57.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:57.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:58.149 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:32:58.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:32:58.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:32:58.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:32:58.320008+0000) 2019-10-18T05:32:58.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:58.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:58.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:58.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:32:59.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:59.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:32:59.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:32:59.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:00.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:00.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:00.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:00.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:01.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:01.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:01.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:01.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:01.783 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:33:01.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:01.782907+0000) 2019-10-18T05:33:02.499 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:33:02.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:02.499235+0000) 2019-10-18T05:33:02.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:02.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:02.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:02.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:03.150 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:03.144+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:03.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:03.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:03.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:03.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:04.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:04.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:04.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:04.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:05.492 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:33:05.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:05.492151+0000) 2019-10-18T05:33:05.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:05.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:05.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:05.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:06.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:06.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:06.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:06.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:07.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:07.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:07.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:07.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:08.151 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:08.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:08.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:33:08.316+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:08.320120+0000) 2019-10-18T05:33:08.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:08.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:08.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:08.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:09.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:09.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:09.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:09.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:10.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:10.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:10.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:10.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:11.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:33:11.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:11.783009+0000) 2019-10-18T05:33:11.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:11.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:11.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:11.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:12.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:33:12.494+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:12.499416+0000) 2019-10-18T05:33:12.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:12.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:12.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:12.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:13.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:13.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:13.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:13.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:13.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:13.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:14.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:14.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:14.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:14.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:15.492 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:33:15.486+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:15.492361+0000) 2019-10-18T05:33:15.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:15.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:15.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:15.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:16.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:16.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:16.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:16.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:17.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:17.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:17.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:17.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:18.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:18.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:18.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:33:18.315+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:18.320235+0000) 2019-10-18T05:33:18.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:18.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:18.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:18.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:19.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:19.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:19.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:20.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:20.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:20.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:20.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:21.784 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:33:21.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:21.783110+0000) 2019-10-18T05:33:21.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:21.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:21.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:21.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:22.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:33:22.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:22.499607+0000) 2019-10-18T05:33:22.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:22.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:22.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:22.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:23.154 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:23.147+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:23.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:23.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:23.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:23.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:24.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:24.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:24.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:24.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:25.493 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:33:25.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:25.492569+0000) 2019-10-18T05:33:25.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:25.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:25.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:25.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:26.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:26.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:26.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:26.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:27.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:27.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:27.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:27.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:28.154 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:28.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:28.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:33:28.315+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:28.320345+0000) 2019-10-18T05:33:28.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:28.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:28.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:28.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:29.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:29.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:29.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:29.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:30.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:30.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:30.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:30.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:31.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:31.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:31.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:31.789 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:33:31.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:31.783208+0000) 2019-10-18T05:33:32.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:33:32.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:32.499810+0000) 2019-10-18T05:33:32.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:32.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:32.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:32.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:33.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:33.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:33.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:33.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:33.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:33.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:34.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:34.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:34.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:34.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:35.493 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:33:35.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:35.492772+0000) 2019-10-18T05:33:35.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:35.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:35.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:35.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:36.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:36.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:36.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:36.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:37.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:37.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:37.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:37.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:38.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:38.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:38.321 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:33:38.315+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:38.320464+0000) 2019-10-18T05:33:38.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:38.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:38.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:38.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:39.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:39.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:39.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:39.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:40.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:40.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:40.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:40.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:41.784 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:33:41.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:41.783308+0000) 2019-10-18T05:33:41.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:41.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:41.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:41.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:42.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:33:42.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:42.499992+0000) 2019-10-18T05:33:42.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:42.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:42.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:42.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:43.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:43.151+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:43.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:43.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:43.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:43.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:44.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:44.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:44.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:44.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:45.493 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:33:45.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:45.492961+0000) 2019-10-18T05:33:45.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:45.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:45.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:45.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:46.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:46.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:46.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:46.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:47.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:47.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:47.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:47.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:48.159 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:48.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:48.322 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:33:48.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:48.320972+0000) 2019-10-18T05:33:48.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:48.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:48.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:48.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:49.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:49.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:49.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:49.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:50.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:50.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:50.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:50.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:51.784 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:33:51.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:51.783409+0000) 2019-10-18T05:33:51.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:51.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:51.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:51.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:52.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:33:52.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:52.500169+0000) 2019-10-18T05:33:52.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:52.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:52.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:52.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:53.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:53.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:53.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:53.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:53.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:53.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:54.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:54.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:54.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:54.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:55.493 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:33:55.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:55.493167+0000) 2019-10-18T05:33:55.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:55.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:55.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:55.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:56.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:56.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:56.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:56.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:57.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:57.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:57.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:57.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:58.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:33:58.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:33:58.322 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:33:58.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:33:58.321094+0000) 2019-10-18T05:33:59.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:59.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:59.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:59.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:33:59.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:59.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:33:59.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:33:59.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:00.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:00.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:00.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:00.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:01.784 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:34:01.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:01.783508+0000) 2019-10-18T05:34:01.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:01.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:01.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:01.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:02.501 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:34:02.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:02.500315+0000) 2019-10-18T05:34:02.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:02.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:02.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:02.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:03.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:03.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:03.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:03.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:03.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:03.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:05.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:05.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:05.493 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:34:05.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:05.493359+0000) 2019-10-18T05:34:06.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:06.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:06.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:06.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:07.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:07.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:07.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:07.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:08.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:08.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:08.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:08.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:08.162 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:08.154+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:08.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:34:08.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:08.321213+0000) 2019-10-18T05:34:09.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:09.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:09.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:09.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:09.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:09.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:09.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:09.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:11.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:11.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:11.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:11.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:11.784 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:34:11.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:11.783606+0000) 2019-10-18T05:34:11.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:11.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:11.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:11.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:12.500 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:34:12.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:12.500492+0000) 2019-10-18T05:34:13.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:13.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:13.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:13.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:13.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:13.158+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:14.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:14.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:14.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:14.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:15.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:14.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:15.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:14.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:15.494 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:34:15.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:15.493568+0000) 2019-10-18T05:34:15.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:15.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:15.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:15.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:17.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:17.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:17.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:17.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:17.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:17.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:17.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:17.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:18.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:18.158+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:18.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:34:18.314+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:18.321332+0000) 2019-10-18T05:34:19.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:19.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:19.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:19.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:20.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:20.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:20.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:20.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:21.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:21.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:21.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:21.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:21.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:34:21.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:21.783703+0000) 2019-10-18T05:34:22.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:22.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:22.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:22.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:22.501 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:34:22.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:22.500689+0000) 2019-10-18T05:34:23.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:23.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:23.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:23.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:23.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:23.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:24.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:24.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:24.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:24.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:25.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:25.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:25.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:25.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:25.494 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:34:25.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:25.493775+0000) 2019-10-18T05:34:26.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:26.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:26.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:26.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:27.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:27.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:27.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:27.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:28.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:28.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:28.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:28.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:28.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:28.157+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:28.322 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:34:28.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:28.321463+0000) 2019-10-18T05:34:29.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:29.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:29.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:29.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:30.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:30.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:30.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:30.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:31.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:31.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:31.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:31.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:31.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:34:31.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:31.783801+0000) 2019-10-18T05:34:32.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:32.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:32.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:32.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:32.501 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:34:32.497+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:32.500868+0000) 2019-10-18T05:34:33.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:33.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:33.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:33.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:33.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:33.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:34.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:33.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:34.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:33.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:34.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:34.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:34.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:34.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:35.494 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:34:35.489+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:35.493960+0000) 2019-10-18T05:34:36.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:36.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:36.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:36.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:36.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:36.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:36.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:36.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:37.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:37.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:37.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:37.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:38.167 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:38.161+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:38.322 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:34:38.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:38.321580+0000) 2019-10-18T05:34:39.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:39.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:39.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:39.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:39.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:39.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:39.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:39.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:40.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:40.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:40.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:40.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:41.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:34:41.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:41.783902+0000) 2019-10-18T05:34:41.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:41.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:41.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:41.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:42.501 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:34:42.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:42.501048+0000) 2019-10-18T05:34:42.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:42.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:42.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:42.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:43.168 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:43.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:43.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:43.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:43.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:43.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:44.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:44.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:44.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:44.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:45.494 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:34:45.492+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:45.494171+0000) 2019-10-18T05:34:45.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:45.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:45.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:45.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:46.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:46.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:46.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:46.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:48.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:48.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:48.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:48.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:48.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:48.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:48.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:34:48.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:48.321698+0000) 2019-10-18T05:34:49.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:49.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:49.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:49.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:50.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:50.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:50.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:50.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:51.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:51.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:51.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:51.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:51.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:34:51.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:51.784000+0000) 2019-10-18T05:34:51.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:51.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:51.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:51.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:52.501 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:34:52.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:52.501226+0000) 2019-10-18T05:34:53.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:52.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:53.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:53.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:53.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:53.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:54.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:54.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:54.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:54.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:55.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:55.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:55.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:55.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:55.494 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:34:55.492+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:55.494383+0000) 2019-10-18T05:34:56.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:56.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:56.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:56.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:57.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:57.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:57.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:57.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:58.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:58.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:58.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:58.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:34:58.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:34:58.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:34:58.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:34:58.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:34:58.321825+0000) 2019-10-18T05:34:59.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:59.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:34:59.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:34:59.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:00.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:00.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:00.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:00.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:01.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:01.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:01.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:01.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:01.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:35:01.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:01.784106+0000) 2019-10-18T05:35:02.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:02.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:02.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:02.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:02.501 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:35:02.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:02.501413+0000) 2019-10-18T05:35:02.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:02.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:02.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:02.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:03.171 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:03.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:04.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:04.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:04.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:04.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:05.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:05.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:05.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:05.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:05.495 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:35:05.492+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:05.494561+0000) 2019-10-18T05:35:05.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:05.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:05.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:05.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:07.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:07.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:07.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:07.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:08.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:08.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:08.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:08.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:08.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:08.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:08.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:35:08.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:08.321945+0000) 2019-10-18T05:35:09.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:09.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:09.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:09.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:10.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:10.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:10.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:10.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:11.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:11.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:11.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:11.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:11.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:35:11.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:11.784208+0000) 2019-10-18T05:35:12.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:12.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:12.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:12.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:12.502 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:35:12.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:12.501598+0000) 2019-10-18T05:35:13.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:13.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:13.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:13.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:13.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:13.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:14.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:14.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:14.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:14.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:15.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:15.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:15.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:15.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:15.496 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:35:15.492+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:15.494776+0000) 2019-10-18T05:35:16.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:16.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:16.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:16.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:17.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:17.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:17.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:17.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:18.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:18.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:18.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:18.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:18.174 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:18.168+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:18.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:35:18.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:18.322069+0000) 2019-10-18T05:35:19.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:19.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:19.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:19.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:20.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:20.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:20.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:20.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:21.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:21.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:21.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:21.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:21.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:35:21.780+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:21.784272+0000) 2019-10-18T05:35:22.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:22.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:22.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:22.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:22.502 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:35:22.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:22.501789+0000) 2019-10-18T05:35:23.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:23.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:23.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:23.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:23.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:23.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:24.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:24.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:24.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:24.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:25.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:25.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:25.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:25.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:25.495 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:35:25.492+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:25.494959+0000) 2019-10-18T05:35:26.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:26.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:26.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:26.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:27.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:27.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:27.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:27.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:28.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:28.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:28.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:28.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:28.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:28.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:28.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:35:28.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:28.322184+0000) 2019-10-18T05:35:29.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:29.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:29.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:29.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:30.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:30.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:30.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:30.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:31.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:31.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:31.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:31.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:31.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:35:31.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:31.784370+0000) 2019-10-18T05:35:32.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:32.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:32.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:32.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:32.502 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:35:32.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:32.501969+0000) 2019-10-18T05:35:33.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:33.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:33.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:33.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:33.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:33.171+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:34.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:34.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:34.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:34.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:35.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:34.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:35.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:34.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:35.495 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:35:35.492+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:35.495103+0000) 2019-10-18T05:35:36.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:35.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:36.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:35.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:37.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:37.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:37.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:37.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:38.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:37.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:38.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:37.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:38.179 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:38.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:38.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:35:38.319+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:38.322311+0000) 2019-10-18T05:35:38.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:38.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:38.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:38.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:40.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:39.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:40.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:39.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:41.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:40.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:41.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:40.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:41.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:35:41.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:41.784469+0000) 2019-10-18T05:35:42.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:42.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:42.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:42.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:42.502 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:35:42.496+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:42.502152+0000) 2019-10-18T05:35:43.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:42.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:43.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:42.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:43.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:43.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:43.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:43.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:43.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:43.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:45.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:45.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:45.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:45.495 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:35:45.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:45.495307+0000) 2019-10-18T05:35:46.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:46.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:46.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:46.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:47.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:47.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:47.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:47.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:48.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:48.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:48.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:48.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:48.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:48.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:48.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:35:48.319+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:48.322429+0000) 2019-10-18T05:35:49.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:49.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:49.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:49.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:50.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:50.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:50.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:50.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:51.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:51.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:51.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:51.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:51.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:35:51.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:51.784570+0000) 2019-10-18T05:35:52.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:52.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:52.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:52.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:52.502 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:35:52.499+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:52.502390+0000) 2019-10-18T05:35:53.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:53.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:53.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:53.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:53.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:53.175+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:54.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:54.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:54.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:54.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:55.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:55.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:55.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:55.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:55.496 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:35:55.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:55.495499+0000) 2019-10-18T05:35:56.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:56.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:56.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:56.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:57.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:57.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:57.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:57.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:58.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:58.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:58.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:58.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:35:58.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:35:58.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:35:58.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:35:58.319+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:35:58.322549+0000) 2019-10-18T05:35:59.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:59.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:35:59.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:35:59.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:00.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:00.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:00.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:00.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:01.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:01.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:01.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:01.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:01.785 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:36:01.779+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:01.784671+0000) 2019-10-18T05:36:02.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:02.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:02.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:02.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:02.503 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:36:02.499+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:02.502608+0000) 2019-10-18T05:36:03.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:03.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:03.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:03.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:03.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:03.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:04.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:04.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:04.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:04.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:05.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:05.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:05.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:05.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:05.496 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:36:05.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:05.495690+0000) 2019-10-18T05:36:06.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:06.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:06.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:06.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:07.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:07.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:07.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:07.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:08.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:08.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:08.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:08.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:08.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:08.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:08.323 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:36:08.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:08.322664+0000) 2019-10-18T05:36:09.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:09.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:09.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:09.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:10.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:10.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:10.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:10.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:11.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:11.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:11.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:11.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:11.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:36:11.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:11.784769+0000) 2019-10-18T05:36:12.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:12.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:12.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:12.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:12.503 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:36:12.499+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:12.502782+0000) 2019-10-18T05:36:13.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:13.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:13.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:13.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:13.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:13.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:14.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:14.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:14.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:14.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:15.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:15.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:15.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:15.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:15.496 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:36:15.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:15.495873+0000) 2019-10-18T05:36:16.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:16.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:16.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:16.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:17.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:17.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:17.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:17.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:18.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:18.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:18.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:18.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:18.186 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:18.178+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:18.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:36:18.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:18.322797+0000) 2019-10-18T05:36:19.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:19.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:19.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:19.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:20.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:20.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:20.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:20.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:21.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:21.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:21.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:21.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:21.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:36:21.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:21.784864+0000) 2019-10-18T05:36:22.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:22.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:22.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:22.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:22.503 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:36:22.499+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:22.502955+0000) 2019-10-18T05:36:23.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:23.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:23.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:23.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:23.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:23.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:24.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:24.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:24.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:24.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:25.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:25.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:25.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:25.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:25.496 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:36:25.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:25.496081+0000) 2019-10-18T05:36:26.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:26.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:26.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:26.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:27.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:27.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:27.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:27.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:28.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:28.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:28.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:28.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:28.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:28.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:28.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:36:28.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:28.322909+0000) 2019-10-18T05:36:29.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:29.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:29.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:29.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:30.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:30.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:30.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:30.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:31.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:31.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:31.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:31.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:31.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:36:31.778+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:31.784963+0000) 2019-10-18T05:36:32.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:32.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:32.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:32.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:32.504 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:36:32.499+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:32.503160+0000) 2019-10-18T05:36:33.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:33.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:33.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:33.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:33.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:33.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:34.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:34.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:34.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:34.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:35.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:35.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:35.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:35.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:35.496 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:36:35.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:35.496289+0000) 2019-10-18T05:36:36.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:36.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:36.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:36.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:37.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:37.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:37.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:37.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:38.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:38.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:38.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:38.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:38.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:38.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:38.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:36:38.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:38.323031+0000) 2019-10-18T05:36:39.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:39.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:39.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:39.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:40.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:40.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:40.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:40.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:41.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:41.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:41.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:41.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:41.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:36:41.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:41.785063+0000) 2019-10-18T05:36:42.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:42.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:42.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:42.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:42.503 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:36:42.499+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:42.503343+0000) 2019-10-18T05:36:43.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:43.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:43.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:43.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:43.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:43.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:44.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:44.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:44.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:44.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:45.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:45.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:45.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:45.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:45.497 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:36:45.491+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:45.496478+0000) 2019-10-18T05:36:46.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:46.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:46.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:46.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:47.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:47.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:47.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:47.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:48.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:48.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:48.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:48.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:48.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:48.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:48.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:36:48.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:48.323145+0000) 2019-10-18T05:36:49.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:49.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:49.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:49.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:50.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:50.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:50.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:50.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:51.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:51.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:51.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:51.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:51.789 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:36:51.777+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:51.785162+0000) 2019-10-18T05:36:52.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:52.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:52.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:52.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:52.504 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:36:52.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:52.503527+0000) 2019-10-18T05:36:53.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:53.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:53.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:53.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:53.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:53.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:54.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:54.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:54.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:54.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:55.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:55.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:55.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:55.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:55.497 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:36:55.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:55.496658+0000) 2019-10-18T05:36:56.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:56.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:56.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:56.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:57.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:57.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:57.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:57.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:58.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:36:58.185+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:36:58.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:58.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:58.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:58.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:36:58.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:36:58.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:36:58.323270+0000) 2019-10-18T05:36:59.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:59.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:36:59.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:36:59.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:00.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:00.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:00.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:00.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:01.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:01.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:01.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:01.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:01.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:37:01.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:01.785258+0000) 2019-10-18T05:37:02.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:02.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:02.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:02.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:02.504 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:37:02.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:02.503737+0000) 2019-10-18T05:37:03.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:03.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:03.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:03.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:03.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:03.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:04.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:04.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:04.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:04.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:05.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:05.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:05.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:05.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:05.500 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:37:05.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:05.496839+0000) 2019-10-18T05:37:06.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:06.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:06.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:06.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:07.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:07.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:07.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:07.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:08.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:08.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:08.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:08.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:08.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:08.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:08.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:37:08.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:08.323386+0000) 2019-10-18T05:37:09.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:09.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:09.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:09.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:10.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:10.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:10.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:10.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:11.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:11.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:11.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:11.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:11.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:37:11.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:11.785359+0000) 2019-10-18T05:37:12.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:12.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:12.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:12.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:12.504 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:37:12.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:12.503941+0000) 2019-10-18T05:37:13.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:13.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:13.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:13.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:13.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:13.193+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:14.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:14.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:15.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:15.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:15.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:15.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:15.497 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:37:15.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:15.497026+0000) 2019-10-18T05:37:16.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:16.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:16.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:16.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:17.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:17.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:17.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:17.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:18.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:17.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:18.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:17.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:18.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:18.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:18.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:37:18.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:18.323507+0000) 2019-10-18T05:37:19.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:19.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:19.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:19.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:20.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:20.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:20.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:20.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:21.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:21.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:21.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:21.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:21.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:37:21.784+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:21.785457+0000) 2019-10-18T05:37:22.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:22.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:22.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:22.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:22.504 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:37:22.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:22.504136+0000) 2019-10-18T05:37:23.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:23.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:23.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:23.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:23.198 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:23.192+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:24.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:24.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:24.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:24.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:25.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:25.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:25.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:25.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:25.497 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:37:25.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:25.497209+0000) 2019-10-18T05:37:25.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:25.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:25.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:25.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:27.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:27.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:27.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:27.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:27.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:27.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:27.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:27.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:28.198 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:28.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:28.324 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:37:28.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:28.323621+0000) 2019-10-18T05:37:29.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:29.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:29.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:29.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:30.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:30.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:30.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:30.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:31.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:31.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:31.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:31.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:31.786 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:37:31.784+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:31.785553+0000) 2019-10-18T05:37:31.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:31.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:31.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:31.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:32.504 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:37:32.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:32.504317+0000) 2019-10-18T05:37:32.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:32.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:32.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:32.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:33.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:33.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:33.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:33.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:33.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:33.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:35.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:35.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:35.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:35.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:35.497 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:37:35.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:35.497388+0000) 2019-10-18T05:37:36.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:36.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:36.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:36.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:37.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:37.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:37.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:37.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:38.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:38.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:38.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:38.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:38.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:38.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:38.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:37:38.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:38.323754+0000) 2019-10-18T05:37:39.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:39.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:39.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:39.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:40.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:40.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:40.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:40.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:41.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:41.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:41.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:41.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:41.789 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:37:41.784+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:41.785653+0000) 2019-10-18T05:37:42.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:42.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:42.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:42.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:42.505 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:37:42.498+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:42.504525+0000) 2019-10-18T05:37:43.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:43.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:43.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:43.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:43.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:43.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:44.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:44.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:44.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:44.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:45.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:45.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:45.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:45.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:45.498 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:37:45.490+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:45.497570+0000) 2019-10-18T05:37:46.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:46.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:46.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:46.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:47.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:47.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:47.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:47.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:48.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:48.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:48.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:48.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:48.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:48.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:48.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:37:48.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:48.323865+0000) 2019-10-18T05:37:49.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:49.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:49.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:49.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:50.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:50.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:50.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:50.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:51.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:51.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:51.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:51.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:51.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:37:51.783+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:51.785753+0000) 2019-10-18T05:37:52.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:52.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:52.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:52.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:52.505 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:37:52.502+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:52.504674+0000) 2019-10-18T05:37:53.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:53.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:53.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:53.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:53.202 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:53.195+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:54.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:54.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:54.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:54.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:55.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:55.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:55.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:55.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:55.498 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:37:55.494+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:55.497748+0000) 2019-10-18T05:37:56.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:56.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:56.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:56.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:57.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:57.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:57.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:57.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:58.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:58.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:58.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:58.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:37:58.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:37:58.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:37:58.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:37:58.319+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:37:58.323993+0000) 2019-10-18T05:37:59.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:59.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:37:59.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:37:59.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:00.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:00.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:00.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:00.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:01.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:01.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:01.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:01.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:01.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:38:01.783+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:01.785852+0000) 2019-10-18T05:38:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:02.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:02.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:02.505 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:38:02.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:02.504858+0000) 2019-10-18T05:38:03.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:03.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:03.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:03.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:03.204 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:03.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:04.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:04.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:04.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:04.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:05.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:05.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:05.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:05.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:05.499 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:38:05.493+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:05.497931+0000) 2019-10-18T05:38:06.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:06.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:06.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:06.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:07.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:07.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:07.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:07.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:08.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:08.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:08.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:08.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:08.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:08.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:08.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:38:08.319+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:08.324129+0000) 2019-10-18T05:38:09.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:09.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:09.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:09.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:10.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:10.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:10.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:10.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:11.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:11.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:11.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:11.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:11.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:38:11.783+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:11.785954+0000) 2019-10-18T05:38:12.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:12.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:12.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:12.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:12.505 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:38:12.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:12.505036+0000) 2019-10-18T05:38:13.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:13.199+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:13.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:13.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:13.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:13.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:14.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:14.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:14.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:14.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:15.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:15.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:15.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:15.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:15.499 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:38:15.493+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:15.498114+0000) 2019-10-18T05:38:16.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:16.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:16.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:16.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:17.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:17.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:17.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:17.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:18.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:18.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:18.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:18.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:18.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:18.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:18.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:38:18.319+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:18.324255+0000) 2019-10-18T05:38:19.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:19.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:19.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:19.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:20.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:20.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:20.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:20.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:21.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:21.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:21.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:21.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:21.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:38:21.783+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:21.786047+0000) 2019-10-18T05:38:22.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:22.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:22.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:22.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:22.505 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:38:22.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:22.505226+0000) 2019-10-18T05:38:23.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:23.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:23.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:23.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:23.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:23.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:24.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:24.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:24.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:24.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:25.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:25.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:25.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:25.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:25.498 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:38:25.497+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:25.498341+0000) 2019-10-18T05:38:26.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:26.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:26.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:26.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:27.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:27.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:27.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:27.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:28.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:28.202+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:28.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:28.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:28.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:28.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:28.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:38:28.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:28.324374+0000) 2019-10-18T05:38:29.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:29.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:29.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:29.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:30.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:30.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:30.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:30.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:31.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:31.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:31.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:31.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:31.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:38:31.782+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:31.786145+0000) 2019-10-18T05:38:32.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:32.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:32.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:32.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:32.505 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:38:32.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:32.505406+0000) 2019-10-18T05:38:33.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:33.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:33.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:33.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:33.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:33.202+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:34.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:34.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:34.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:34.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:35.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:35.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:35.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:35.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:35.499 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:38:35.497+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:35.498520+0000) 2019-10-18T05:38:36.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:36.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:36.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:36.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:37.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:37.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:37.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:37.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:38.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:38.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:38.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:38.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:38.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:38.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:38.325 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:38:38.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:38.324502+0000) 2019-10-18T05:38:39.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:39.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:39.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:39.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:40.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:40.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:40.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:40.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:41.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:41.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:41.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:41.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:41.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:38:41.782+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:41.786245+0000) 2019-10-18T05:38:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:42.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:42.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:42.506 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:38:42.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:42.505585+0000) 2019-10-18T05:38:43.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:43.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:43.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:43.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:43.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:43.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:44.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:44.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:44.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:44.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:45.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:45.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:45.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:45.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:45.499 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:38:45.497+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:45.498671+0000) 2019-10-18T05:38:46.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:46.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:46.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:46.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:47.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:47.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:47.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:47.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:48.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:48.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:48.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:48.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:48.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:48.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:48.326 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:38:48.318+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:48.324622+0000) 2019-10-18T05:38:49.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:49.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:49.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:49.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:50.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:50.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:50.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:50.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:51.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:51.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:51.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:51.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:51.789 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:38:51.782+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:51.786343+0000) 2019-10-18T05:38:52.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:52.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:52.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:52.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:52.506 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:38:52.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:52.505759+0000) 2019-10-18T05:38:53.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:53.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:53.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:53.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:53.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:53.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:54.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:54.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:54.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:54.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:55.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:55.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:55.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:55.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:55.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:38:55.497+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:55.498879+0000) 2019-10-18T05:38:56.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:56.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:56.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:56.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:57.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:57.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:57.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:57.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:58.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:58.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:58.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:58.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:38:58.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:38:58.205+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:38:58.326 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:38:58.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:38:58.324761+0000) 2019-10-18T05:38:59.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:59.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:38:59.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:38:59.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:00.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:00.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:00.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:00.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:01.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:01.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:01.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:01.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:01.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:39:01.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:01.786443+0000) 2019-10-18T05:39:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:02.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:02.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:02.506 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:39:02.501+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:02.505909+0000) 2019-10-18T05:39:03.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:03.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:03.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:03.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:03.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:03.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:04.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:04.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:04.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:04.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:05.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:05.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:05.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:05.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:05.499 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:39:05.497+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:05.499061+0000) 2019-10-18T05:39:06.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:06.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:06.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:06.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:07.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:07.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:07.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:07.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:08.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:08.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:08.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:08.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:08.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:08.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:08.326 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:39:08.317+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:08.324882+0000) 2019-10-18T05:39:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:09.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:09.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:10.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:10.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:10.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:10.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:11.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:11.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:11.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:11.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:11.787 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:39:11.781+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:11.786549+0000) 2019-10-18T05:39:12.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:12.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:12.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:12.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:12.507 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:39:12.500+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:12.506088+0000) 2019-10-18T05:39:13.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:13.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:13.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:13.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:13.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:13.209+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:14.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:14.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:14.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:14.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:15.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:15.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:15.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:15.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:15.499 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:39:15.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:15.499243+0000) 2019-10-18T05:39:16.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:16.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:16.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:16.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:17.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:17.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:17.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:17.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:18.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:18.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:18.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:18.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:18.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:18.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:18.326 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:39:18.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:18.325025+0000) 2019-10-18T05:39:19.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:19.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:19.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:19.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:20.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:20.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:20.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:20.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:21.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:21.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:21.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:21.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:21.788 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:39:21.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:21.786655+0000) 2019-10-18T05:39:22.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:22.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:22.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:22.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:22.506 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:39:22.504+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:22.506277+0000) 2019-10-18T05:39:23.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:23.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:23.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:23.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:23.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:23.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:24.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:24.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:24.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:24.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:25.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:25.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:25.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:25.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:25.500 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:39:25.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:25.499427+0000) 2019-10-18T05:39:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:26.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:26.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:26.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:27.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:27.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:27.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:27.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:28.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:28.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:28.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:28.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:28.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:28.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:28.326 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:39:28.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:28.325165+0000) 2019-10-18T05:39:29.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:29.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:29.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:29.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:30.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:30.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:30.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:30.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:31.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:31.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:31.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:31.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:31.788 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:39:31.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:31.786758+0000) 2019-10-18T05:39:32.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:32.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:32.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:32.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:32.507 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:39:32.504+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:32.506456+0000) 2019-10-18T05:39:33.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:33.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:28:58.959934+0000 (oldest deadline 2019-10-18T05:29:18.959934+0000) 2019-10-18T05:39:33.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:33.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:33.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:33.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:34.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:34.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:35.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:35.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:35.500 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:39:35.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:35.499637+0000) 2019-10-18T05:39:36.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:36.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:37.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:37.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:38.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:38.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:38.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:38.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:38.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:39:38.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:38.325298+0000) 2019-10-18T05:39:39.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:39.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:40.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:40.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:41.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:41.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:41.788 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:39:41.784+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:41.786867+0000) 2019-10-18T05:39:42.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:42.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:42.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:39:42.504+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:42.506632+0000) 2019-10-18T05:39:43.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:43.216+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:43.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:43.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:44.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:44.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:45.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:45.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:45.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:39:45.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:45.499808+0000) 2019-10-18T05:39:46.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:46.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:47.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:47.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:48.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:48.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:48.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:48.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:48.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:39:48.320+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:48.325413+0000) 2019-10-18T05:39:49.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:49.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:50.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:50.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:51.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:51.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:51.790 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:39:51.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:51.789502+0000) 2019-10-18T05:39:52.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:52.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:52.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:39:52.504+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:52.506845+0000) 2019-10-18T05:39:53.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:53.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:53.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:53.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:54.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:54.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:55.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:55.500 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:39:55.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:55.500022+0000) 2019-10-18T05:39:56.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:56.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:39:56.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:56.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:57.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:39:57.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:57.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:58.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:39:58.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:39:58.326 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:39:58.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:39:58.325538+0000) 2019-10-18T05:39:58.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:58.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:39:58.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:58.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:39:59.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:59.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:39:59.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:39:59.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:00.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:00.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:00.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:00.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:01.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:01.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:01.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:01.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:01.790 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:40:01.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:01.789604+0000) 2019-10-18T05:40:02.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:02.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:02.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:02.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:02.507 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:40:02.504+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:02.507056+0000) 2019-10-18T05:40:03.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:03.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:03.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:03.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:03.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:03.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:04.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:04.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:04.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:04.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:05.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:05.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:05.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:05.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:05.500 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:40:05.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:05.500216+0000) 2019-10-18T05:40:06.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:06.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:06.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:06.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:07.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:07.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:07.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:07.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:08.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:08.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:08.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:08.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:08.227 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:08.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:08.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:40:08.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:08.325661+0000) 2019-10-18T05:40:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:09.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:09.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:10.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:10.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:10.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:10.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:11.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:11.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:11.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:11.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:11.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:40:11.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:11.789706+0000) 2019-10-18T05:40:12.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:12.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:12.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:12.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:12.507 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:40:12.504+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:12.507243+0000) 2019-10-18T05:40:13.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:13.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:13.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:13.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:13.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:13.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:14.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:14.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:14.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:14.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:15.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:15.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:15.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:15.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:15.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:40:15.496+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:15.500414+0000) 2019-10-18T05:40:16.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:16.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:16.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:16.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:17.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:17.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:17.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:17.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:18.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:18.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:18.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:18.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:18.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:18.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:18.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:40:18.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:18.325794+0000) 2019-10-18T05:40:19.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:19.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:19.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:19.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:20.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:20.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:20.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:20.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:21.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:21.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:21.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:21.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:21.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:40:21.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:21.789798+0000) 2019-10-18T05:40:22.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:22.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:22.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:22.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:22.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:40:22.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:22.507456+0000) 2019-10-18T05:40:23.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:23.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:23.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:23.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:23.229 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:23.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:24.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:24.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:24.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:24.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:25.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:25.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:25.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:25.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:25.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:40:25.495+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:25.500595+0000) 2019-10-18T05:40:26.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:26.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:26.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:26.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:27.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:27.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:28.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:28.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:28.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:28.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:28.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:28.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:28.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:40:28.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:28.325913+0000) 2019-10-18T05:40:29.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:29.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:29.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:29.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:30.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:30.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:30.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:30.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:31.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:31.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:31.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:31.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:31.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:40:31.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:31.789899+0000) 2019-10-18T05:40:32.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:32.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:32.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:32.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:32.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:40:32.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:32.507640+0000) 2019-10-18T05:40:33.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:33.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:33.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:33.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:33.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:33.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:34.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:34.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:34.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:35.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:35.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:35.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:35.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:35.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:40:35.495+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:35.500806+0000) 2019-10-18T05:40:36.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:36.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:36.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:36.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:37.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:37.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:37.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:37.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:38.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:38.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:38.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:38.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:38.233 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:38.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:38.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:40:38.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:38.326041+0000) 2019-10-18T05:40:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:39.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:39.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:40.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:40.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:40.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:40.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:41.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:41.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:41.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:41.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:41.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:40:41.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:41.790005+0000) 2019-10-18T05:40:42.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:42.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:42.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:42.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:42.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:40:42.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:42.507819+0000) 2019-10-18T05:40:43.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:43.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:43.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:43.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:43.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:43.226+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:44.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:44.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:44.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:44.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:45.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:45.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:45.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:45.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:45.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:40:45.495+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:45.500989+0000) 2019-10-18T05:40:46.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:46.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:46.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:46.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:47.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:47.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:47.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:47.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:48.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:48.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:48.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:48.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:48.234 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:48.226+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:48.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:40:48.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:48.326161+0000) 2019-10-18T05:40:49.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:49.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:49.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:49.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:50.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:50.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:50.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:50.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:51.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:51.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:51.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:51.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:51.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:40:51.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:51.790098+0000) 2019-10-18T05:40:52.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:52.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:52.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:52.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:52.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:40:52.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:52.508008+0000) 2019-10-18T05:40:53.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:53.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:53.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:53.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:53.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:53.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:54.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:54.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:54.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:54.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:55.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:55.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:55.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:55.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:55.501 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:40:55.495+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:55.501171+0000) 2019-10-18T05:40:56.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:56.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:56.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:56.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:57.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:57.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:57.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:57.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:58.235 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:40:58.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:40:58.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:40:58.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:40:58.326304+0000) 2019-10-18T05:40:58.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:58.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:58.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:58.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:40:59.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:59.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:40:59.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:40:59.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:00.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:00.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:00.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:01.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:01.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:01.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:01.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:01.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:41:01.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:01.790196+0000) 2019-10-18T05:41:02.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:02.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:02.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:02.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:02.508 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:41:02.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:02.508195+0000) 2019-10-18T05:41:03.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:03.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:03.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:03.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:03.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:03.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:04.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:04.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:04.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:04.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:05.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:05.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:05.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:05.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:05.502 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:41:05.495+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:05.501337+0000) 2019-10-18T05:41:06.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:06.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:06.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:06.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:07.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:07.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:07.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:07.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:08.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:08.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:08.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:08.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:08.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:08.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:08.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:41:08.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:08.326426+0000) 2019-10-18T05:41:09.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:09.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:09.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:09.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:10.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:10.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:10.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:10.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:11.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:11.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:11.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:11.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:11.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:41:11.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:11.790296+0000) 2019-10-18T05:41:12.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:12.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:12.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:12.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:12.509 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:41:12.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:12.508383+0000) 2019-10-18T05:41:13.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:13.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:13.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:13.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:13.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:13.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:14.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:14.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:14.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:14.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:15.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:15.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:15.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:15.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:15.503 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:41:15.495+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:15.501499+0000) 2019-10-18T05:41:16.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:16.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:16.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:16.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:17.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:17.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:17.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:17.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:18.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:18.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:18.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:18.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:18.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:18.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:18.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:41:18.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:18.326559+0000) 2019-10-18T05:41:19.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:19.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:19.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:19.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:20.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:20.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:20.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:20.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:21.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:21.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:21.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:21.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:21.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:41:21.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:21.790390+0000) 2019-10-18T05:41:22.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:22.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:22.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:22.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:22.509 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:41:22.503+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:22.508579+0000) 2019-10-18T05:41:23.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:23.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:23.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:23.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:23.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:23.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:24.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:24.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:25.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:25.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:25.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:25.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:25.502 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:41:25.494+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:25.501676+0000) 2019-10-18T05:41:26.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:26.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:26.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:26.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:27.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:27.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:27.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:27.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:28.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:28.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:28.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:28.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:28.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:28.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:28.327 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:41:28.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:28.326684+0000) 2019-10-18T05:41:29.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:29.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:29.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:29.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:30.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:30.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:30.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:30.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:31.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:31.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:31.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:31.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:31.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:41:31.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:31.790487+0000) 2019-10-18T05:41:32.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:32.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:32.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:32.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:32.509 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:41:32.502+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:32.508764+0000) 2019-10-18T05:41:33.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:33.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:33.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:33.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:33.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:33.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:34.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:34.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:34.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:34.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:35.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:35.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:35.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:35.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:35.502 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:41:35.494+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:35.501890+0000) 2019-10-18T05:41:36.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:36.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:36.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:36.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:37.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:37.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:37.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:37.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:38.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:38.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:38.328 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:41:38.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:38.326827+0000) 2019-10-18T05:41:38.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:38.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:38.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:38.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:39.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:39.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:39.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:39.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:40.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:40.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:40.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:40.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:41.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:41.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:41.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:41.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:41.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:41:41.789+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:41.790586+0000) 2019-10-18T05:41:42.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:42.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:42.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:42.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:42.510 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:41:42.502+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:42.508910+0000) 2019-10-18T05:41:43.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:43.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:43.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:43.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:43.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:43.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:44.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:44.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:44.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:44.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:45.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:45.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:45.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:45.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:45.502 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:41:45.494+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:45.502098+0000) 2019-10-18T05:41:46.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:46.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:46.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:46.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:47.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:47.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:47.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:47.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:48.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:48.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:48.328 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:41:48.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:48.326968+0000) 2019-10-18T05:41:48.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:48.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:48.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:48.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:49.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:49.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:49.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:49.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:50.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:50.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:50.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:50.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:51.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:51.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:51.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:51.791 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:41:51.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:51.790687+0000) 2019-10-18T05:41:52.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:52.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:52.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:52.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:52.509 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:41:52.502+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:52.509092+0000) 2019-10-18T05:41:53.245 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:53.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:53.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:53.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:53.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:53.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:54.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:54.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:54.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:54.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 468 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:55.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:55.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:55.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:55.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:55.502 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:41:55.498+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:55.502282+0000) 2019-10-18T05:41:56.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:56.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:56.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:56.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:57.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:57.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:57.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:57.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:58.247 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:41:58.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:41:58.328 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:41:58.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:41:58.327097+0000) 2019-10-18T05:41:58.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:58.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:58.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:58.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:41:59.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:59.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:41:59.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:41:59.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:00.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:00.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:00.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:00.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:01.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:01.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:01.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:01.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:01.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:42:01.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:01.790787+0000) 2019-10-18T05:42:02.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:02.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:02.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:02.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:02.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:42:02.502+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:02.509284+0000) 2019-10-18T05:42:03.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:03.240+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:03.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:03.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:03.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:03.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:04.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:04.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:04.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:04.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:05.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:05.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:05.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:05.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:05.503 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:42:05.498+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:05.502462+0000) 2019-10-18T05:42:06.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:06.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:06.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:06.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:07.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:07.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:07.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:07.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:08.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:08.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:08.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:08.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:08.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:08.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:08.328 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:42:08.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:08.327215+0000) 2019-10-18T05:42:09.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:09.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:09.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:09.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:10.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:10.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:10.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:10.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:11.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:11.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:11.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:11.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:11.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:42:11.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:11.790886+0000) 2019-10-18T05:42:12.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:12.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:12.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:12.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:12.510 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:42:12.502+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:12.509445+0000) 2019-10-18T05:42:13.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:13.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:13.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:13.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:13.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:13.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:14.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:14.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:14.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:14.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:15.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:15.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:15.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:15.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:15.503 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:42:15.498+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:15.502668+0000) 2019-10-18T05:42:16.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:16.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:16.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:16.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:17.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:17.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:17.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:17.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:18.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:18.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:18.328 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:42:18.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:18.327343+0000) 2019-10-18T05:42:18.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:18.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:18.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:18.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:19.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:19.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:19.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:19.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:20.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:20.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:20.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:20.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:21.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:21.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:21.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:21.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:21.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:42:21.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:21.790981+0000) 2019-10-18T05:42:22.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:22.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:22.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:22.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:22.510 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:42:22.506+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:22.509621+0000) 2019-10-18T05:42:23.250 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:23.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:23.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:23.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:23.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:23.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:24.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:24.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:24.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:24.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:25.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:25.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:25.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:25.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:25.503 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:42:25.502+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:25.502860+0000) 2019-10-18T05:42:26.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:26.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:26.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:26.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:27.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:27.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:27.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:27.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:28.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:28.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:28.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:28.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:28.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:28.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:28.328 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:42:28.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:28.327452+0000) 2019-10-18T05:42:29.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:29.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:29.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:29.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:30.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:30.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:30.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:30.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:31.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:31.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:31.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:31.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:31.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:42:31.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:31.791083+0000) 2019-10-18T05:42:32.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:32.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:32.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:32.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:32.512 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:42:32.505+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:32.509804+0000) 2019-10-18T05:42:33.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:33.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:33.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:33.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:33.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:33.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:34.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:34.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:34.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:34.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:35.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:35.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:35.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:35.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:35.503 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:42:35.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:35.503069+0000) 2019-10-18T05:42:36.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:36.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:36.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:36.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:37.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:37.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:37.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:37.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:38.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:38.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:38.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:38.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:38.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:38.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:38.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:42:38.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:38.327527+0000) 2019-10-18T05:42:39.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:39.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:39.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:39.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:40.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:40.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:40.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:40.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:41.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:41.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:41.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:41.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:41.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:42:41.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:41.791182+0000) 2019-10-18T05:42:42.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:42.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:42.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:42.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:42.510 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:42:42.505+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:42.509988+0000) 2019-10-18T05:42:43.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:43.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:43.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:43.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:43.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:43.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:44.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:44.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:44.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:44.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:45.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:45.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:45.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:45.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:45.503 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:42:45.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:45.503217+0000) 2019-10-18T05:42:46.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:46.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:46.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:46.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:47.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:47.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:47.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:47.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:48.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:48.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:48.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:48.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:48.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:48.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:48.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:42:48.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:48.327636+0000) 2019-10-18T05:42:49.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:49.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:49.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:49.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:50.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:50.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:50.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:50.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:51.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:51.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:51.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:51.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:51.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:42:51.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:51.791278+0000) 2019-10-18T05:42:52.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:52.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:52.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:52.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:52.510 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:42:52.505+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:52.510130+0000) 2019-10-18T05:42:53.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:53.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:53.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:53.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:53.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:53.251+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:54.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:54.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:54.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:54.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:55.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:55.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:55.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:55.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:55.504 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:42:55.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:55.503424+0000) 2019-10-18T05:42:56.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:56.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:56.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:56.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:57.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:57.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:57.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:57.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:58.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:58.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:58.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:58.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:42:58.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:42:58.251+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:42:58.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:42:58.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:42:58.327744+0000) 2019-10-18T05:42:59.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:59.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:42:59.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:42:59.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:00.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:00.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:00.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:00.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:01.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:01.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:01.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:01.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:01.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:43:01.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:01.791376+0000) 2019-10-18T05:43:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:02.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:02.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:02.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:43:02.509+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:02.510339+0000) 2019-10-18T05:43:03.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:03.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:03.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:03.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:03.257 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:03.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:04.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:04.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:04.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:04.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:05.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:05.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:05.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:05.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:05.504 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:43:05.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:05.503609+0000) 2019-10-18T05:43:06.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:06.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:06.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:06.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:07.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:07.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:07.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:07.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:08.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:08.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:08.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:08.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:08.257 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:08.250+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:08.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:43:08.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:08.327853+0000) 2019-10-18T05:43:09.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:09.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:09.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:10.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:10.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:10.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:10.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:11.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:11.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:11.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:11.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:11.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:43:11.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:11.791479+0000) 2019-10-18T05:43:12.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:12.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:12.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:12.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:12.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:43:12.509+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:12.510486+0000) 2019-10-18T05:43:13.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:13.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:13.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:13.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:13.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:13.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:14.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:14.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:14.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:14.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:15.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:15.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:15.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:15.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:15.504 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:43:15.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:15.503797+0000) 2019-10-18T05:43:16.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:16.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:16.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:16.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:17.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:17.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:17.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:17.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:18.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:18.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:18.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:18.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:18.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:18.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:18.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:43:18.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:18.327964+0000) 2019-10-18T05:43:19.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:19.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:19.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:19.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:20.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:20.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:20.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:20.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:21.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:21.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:21.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:21.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:21.792 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:43:21.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:21.791576+0000) 2019-10-18T05:43:22.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:22.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:22.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:22.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:22.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:43:22.509+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:22.510692+0000) 2019-10-18T05:43:23.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:23.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:23.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:23.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:23.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:23.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:24.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:24.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:24.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:24.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:25.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:25.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:25.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:25.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:25.504 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:43:25.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:25.503952+0000) 2019-10-18T05:43:26.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:26.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:26.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:26.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:27.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:27.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:27.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:27.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:28.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:28.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:28.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:28.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:28.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:28.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:28.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:43:28.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:28.328075+0000) 2019-10-18T05:43:29.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:29.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:29.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:29.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:30.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:30.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:30.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:30.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:31.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:31.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:31.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:31.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:31.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:43:31.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:31.791671+0000) 2019-10-18T05:43:32.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:31.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:32.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:31.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:32.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:43:32.509+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:32.510869+0000) 2019-10-18T05:43:33.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:33.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:33.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:33.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:33.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:33.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:34.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:34.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:34.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:34.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:35.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:35.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:35.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:35.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:35.504 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:43:35.501+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:35.504167+0000) 2019-10-18T05:43:36.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:36.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:36.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:36.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:37.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:37.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:37.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:37.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:38.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:38.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:38.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:38.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:38.263 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:38.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:38.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:43:38.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:38.328187+0000) 2019-10-18T05:43:39.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:39.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:39.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:39.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:40.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:40.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:40.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:40.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:41.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:41.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:41.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:41.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:41.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:43:41.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:41.791767+0000) 2019-10-18T05:43:42.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:42.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:42.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:42.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:42.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:43:42.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:42.511057+0000) 2019-10-18T05:43:43.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:43.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:43.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:43.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:43.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:43.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:44.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:44.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:44.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:44.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:45.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:45.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:45.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:45.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:45.505 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:43:45.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:45.504378+0000) 2019-10-18T05:43:46.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:46.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:46.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:46.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:47.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:47.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:47.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:47.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:48.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:48.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:48.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:48.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:48.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:48.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:48.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:43:48.321+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:48.328297+0000) 2019-10-18T05:43:49.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:49.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:49.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:49.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:50.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:50.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:50.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:50.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:51.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:51.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:51.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:51.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:51.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:43:51.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:51.791865+0000) 2019-10-18T05:43:52.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:52.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:52.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:52.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:52.511 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:43:52.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:52.511197+0000) 2019-10-18T05:43:53.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:53.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:53.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:53.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:53.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:53.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:54.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:54.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:54.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:54.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:55.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:55.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:55.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:55.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:55.506 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:43:55.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:55.504587+0000) 2019-10-18T05:43:56.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:56.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:56.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:56.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:57.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:57.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:57.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:57.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:58.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:43:58.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:43:58.332 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:43:58.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:43:58.328409+0000) 2019-10-18T05:43:58.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:58.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:58.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:58.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:43:59.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:59.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:43:59.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:43:59.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:00.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:00.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:00.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:00.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:01.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:01.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:01.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:01.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:01.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:44:01.789+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:01.791964+0000) 2019-10-18T05:44:02.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:02.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:02.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:02.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:02.512 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:44:02.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:02.511409+0000) 2019-10-18T05:44:03.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:03.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:03.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:03.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:03.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:03.265+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:04.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:04.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:04.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:04.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:05.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:05.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:05.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:05.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:05.505 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:44:05.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:05.504779+0000) 2019-10-18T05:44:06.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:06.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:06.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:06.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:07.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:07.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:07.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:07.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:08.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:08.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:08.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:08.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:08.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:08.265+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:08.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:44:08.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:08.328519+0000) 2019-10-18T05:44:09.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:09.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:09.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:09.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:10.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:10.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:10.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:10.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:11.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:11.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:11.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:11.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:11.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:44:11.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:11.792063+0000) 2019-10-18T05:44:12.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:12.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:12.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:12.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:12.512 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:44:12.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:12.511598+0000) 2019-10-18T05:44:13.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:13.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:13.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:13.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:13.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:13.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:14.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:14.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:14.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:14.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:15.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:15.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:15.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:15.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:15.506 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:44:15.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:15.504993+0000) 2019-10-18T05:44:16.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:16.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:16.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:16.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:17.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:17.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:17.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:17.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:18.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:18.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:18.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:18.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:18.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:18.264+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:18.329 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:44:18.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:18.328628+0000) 2019-10-18T05:44:19.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:19.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:19.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:19.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:20.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:20.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:20.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:20.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:21.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:21.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:21.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:21.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:21.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:44:21.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:21.792169+0000) 2019-10-18T05:44:22.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:22.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:22.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:22.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:22.512 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:44:22.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:22.511808+0000) 2019-10-18T05:44:23.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:23.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:23.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:23.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:23.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:23.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:24.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:24.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:24.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:24.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:25.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:25.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:25.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:25.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:25.505 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:44:25.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:25.505190+0000) 2019-10-18T05:44:26.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:26.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:26.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:26.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:27.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:27.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:27.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:27.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:28.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:28.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:28.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:28.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:28.272 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:28.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:28.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:44:28.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:28.328740+0000) 2019-10-18T05:44:29.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:29.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:29.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:29.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:30.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:30.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:30.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:30.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:31.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:31.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:31.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:31.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:31.799 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:44:31.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:31.792273+0000) 2019-10-18T05:44:32.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:32.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:32.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:32.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:32.513 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:44:32.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:32.512000+0000) 2019-10-18T05:44:33.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:33.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:33.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:33.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:33.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:33.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:34.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:34.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:34.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:35.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:35.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:35.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:35.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:35.506 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:44:35.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:35.505383+0000) 2019-10-18T05:44:36.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:36.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:36.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:36.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:37.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:37.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:37.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:37.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:38.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:38.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:38.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:38.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:38.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:38.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:38.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:44:38.324+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:38.328851+0000) 2019-10-18T05:44:39.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:39.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:39.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:39.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:40.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:40.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:40.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:40.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:41.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:41.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:41.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:41.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:41.793 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:44:41.788+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:41.792375+0000) 2019-10-18T05:44:42.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:42.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:42.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:42.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:42.512 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:44:42.508+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:42.512179+0000) 2019-10-18T05:44:43.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:43.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:43.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:43.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:43.274 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:43.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:44.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:44.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:44.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:44.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:45.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:45.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:45.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:45.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:45.506 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:44:45.500+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:45.505595+0000) 2019-10-18T05:44:46.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:46.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:46.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:46.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:47.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:47.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:47.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:47.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:48.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:48.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:48.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:48.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:48.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:48.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:48.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:44:48.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:48.328959+0000) 2019-10-18T05:44:49.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:49.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:49.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:49.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:50.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:50.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:50.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:50.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:51.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:51.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:51.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:51.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:51.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:44:51.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:51.792480+0000) 2019-10-18T05:44:52.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:52.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:52.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:52.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:52.513 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:44:52.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:52.512363+0000) 2019-10-18T05:44:53.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:53.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:53.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:53.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:53.276 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:53.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:53.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:53.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:53.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:53.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:55.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:55.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:55.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:55.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:55.507 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:44:55.499+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:55.505780+0000) 2019-10-18T05:44:55.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:55.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:55.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:55.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:56.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:56.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:56.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:56.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:57.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:57.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:57.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:57.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:58.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:44:58.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:44:58.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:44:58.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:44:58.329067+0000) 2019-10-18T05:44:59.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:59.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:59.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:59.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:44:59.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:59.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:44:59.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:44:59.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:00.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:00.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:00.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:00.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:01.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:45:01.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:01.792577+0000) 2019-10-18T05:45:01.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:01.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:01.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:01.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:02.513 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:45:02.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:02.512580+0000) 2019-10-18T05:45:02.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:02.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:02.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:02.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:03.278 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:03.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:03.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:03.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:03.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:03.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:04.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:04.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:04.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:04.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:05.506 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:45:05.499+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:05.505964+0000) 2019-10-18T05:45:05.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:05.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:05.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:05.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:06.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:06.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:06.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:06.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:07.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:07.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:07.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:07.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:08.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:08.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:08.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:45:08.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:08.329173+0000) 2019-10-18T05:45:08.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:08.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:08.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:08.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:09.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:09.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:09.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:09.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:10.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:10.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:10.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:10.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:11.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:45:11.787+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:11.792680+0000) 2019-10-18T05:45:11.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:11.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:11.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:11.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:12.513 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:45:12.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:12.512789+0000) 2019-10-18T05:45:12.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:12.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:12.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:12.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:13.280 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:13.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:13.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:13.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:13.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:13.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:14.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:14.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:14.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:14.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:15.508 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:45:15.503+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:15.506201+0000) 2019-10-18T05:45:15.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:15.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:15.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:15.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:16.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:16.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:16.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:16.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:17.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:17.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:17.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:17.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:18.280 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:18.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:18.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:45:18.323+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:18.329268+0000) 2019-10-18T05:45:18.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:18.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:18.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:18.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:19.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:19.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:19.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:19.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:20.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:20.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:20.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:20.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:21.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:45:21.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:21.792778+0000) 2019-10-18T05:45:21.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:21.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:21.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:21.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:22.514 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:45:22.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:22.512975+0000) 2019-10-18T05:45:22.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:22.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:22.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:22.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:23.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:23.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:23.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:23.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:23.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:23.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:24.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:24.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:24.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:24.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:25.507 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:45:25.503+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:25.506396+0000) 2019-10-18T05:45:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:25.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:25.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:25.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:26.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:26.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:26.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:26.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:27.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:27.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:27.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:27.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:28.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:28.274+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:28.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:45:28.322+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:28.329375+0000) 2019-10-18T05:45:28.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:28.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:28.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:28.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:29.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:29.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:29.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:29.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:30.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:30.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:30.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:30.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:31.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:45:31.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:31.792876+0000) 2019-10-18T05:45:31.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:31.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:31.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:31.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:32.513 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:45:32.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:32.513168+0000) 2019-10-18T05:45:32.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:32.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:32.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:32.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:33.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:33.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:33.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:33.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:33.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:33.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:34.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:34.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:34.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:34.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:35.507 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:45:35.503+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:35.506586+0000) 2019-10-18T05:45:35.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:35.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:35.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:35.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:36.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:36.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:36.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:36.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:37.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:37.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:37.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:37.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:38.284 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:38.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:38.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:45:38.326+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:38.329497+0000) 2019-10-18T05:45:38.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:38.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:38.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:38.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:39.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:39.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:39.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:39.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:40.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:40.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:40.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:40.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:41.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:45:41.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:41.792974+0000) 2019-10-18T05:45:41.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:41.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:41.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:41.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:42.514 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:45:42.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:42.513349+0000) 2019-10-18T05:45:42.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:42.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:42.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:42.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:43.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:43.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:43.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:43.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:43.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:43.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:44.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:44.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:44.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:44.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:45.508 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:45:45.503+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:45.506800+0000) 2019-10-18T05:45:45.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:45.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:45.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:45.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:46.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:46.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:46.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:46.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:47.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:47.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:47.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:47.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:48.289 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:48.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:48.330 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:45:48.326+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:48.329605+0000) 2019-10-18T05:45:48.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:48.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:48.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:48.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:49.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:49.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:49.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:49.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:50.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:50.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:50.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:50.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:51.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:45:51.786+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:51.793073+0000) 2019-10-18T05:45:51.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:51.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:51.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:51.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:52.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:45:52.507+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:52.513537+0000) 2019-10-18T05:45:52.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:52.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:52.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:52.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:53.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:53.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:53.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:53.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:53.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:53.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:54.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:54.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:54.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:54.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:55.507 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:45:55.503+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:55.506978+0000) 2019-10-18T05:45:55.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:55.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:55.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:55.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:56.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:56.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:56.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:56.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:57.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:57.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:57.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:57.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:58.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:45:58.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:45:58.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:45:58.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:45:58.329714+0000) 2019-10-18T05:45:58.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:58.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:58.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:58.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:45:59.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:59.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:45:59.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:45:59.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:00.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:00.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:00.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:00.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:01.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:01.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:01.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:01.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:01.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:46:01.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:01.793176+0000) 2019-10-18T05:46:02.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:46:02.506+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:02.513718+0000) 2019-10-18T05:46:02.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:02.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:02.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:02.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:03.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:03.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:03.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:03.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:03.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:03.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:04.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:04.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:04.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:04.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:05.507 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:46:05.502+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:05.507193+0000) 2019-10-18T05:46:05.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:05.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:05.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:05.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:06.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:06.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:06.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:06.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:07.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:07.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:07.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:07.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:08.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:08.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:08.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:46:08.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:08.329838+0000) 2019-10-18T05:46:08.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:08.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:08.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:08.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:09.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:09.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:09.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:09.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:10.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:10.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:10.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:10.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:11.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:11.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:11.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:11.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:11.794 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:46:11.785+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:11.793280+0000) 2019-10-18T05:46:12.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:12.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:12.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:12.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:12.514 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:46:12.506+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:12.513931+0000) 2019-10-18T05:46:13.293 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:13.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:13.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:13.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:13.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:13.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:14.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:14.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:14.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:14.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:15.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:15.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:15.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:15.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:15.508 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:46:15.502+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:15.507404+0000) 2019-10-18T05:46:16.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:16.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:16.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:16.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:17.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:17.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:17.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:17.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:18.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:18.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:18.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:46:18.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:18.329981+0000) 2019-10-18T05:46:18.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:18.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:18.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:18.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:19.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:19.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:19.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:19.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:20.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:20.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:20.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:20.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:21.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:21.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:21.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:21.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:21.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:46:21.789+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:21.793374+0000) 2019-10-18T05:46:22.514 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:46:22.506+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:22.514110+0000) 2019-10-18T05:46:22.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:22.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:22.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:22.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:23.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:23.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:23.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:23.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:23.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:23.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:24.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:24.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:24.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:24.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:25.508 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:46:25.502+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:25.507589+0000) 2019-10-18T05:46:25.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:25.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:25.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:25.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:26.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:26.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:26.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:26.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:27.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:27.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:27.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:27.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:28.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:28.293+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:28.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:46:28.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:28.330111+0000) 2019-10-18T05:46:28.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:28.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:28.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:28.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:29.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:29.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:29.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:29.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:30.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:30.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:30.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:30.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:31.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:31.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:31.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:31.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:31.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:46:31.792+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:31.793473+0000) 2019-10-18T05:46:32.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:32.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:32.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:32.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:32.515 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:46:32.510+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:32.514344+0000) 2019-10-18T05:46:33.301 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:33.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:33.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:33.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:33.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:33.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:34.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:34.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:34.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:34.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:35.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:35.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:35.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:35.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:35.509 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:46:35.502+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:35.507772+0000) 2019-10-18T05:46:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:36.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:36.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:36.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:37.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:37.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:37.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:37.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:38.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:38.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:38.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:46:38.328+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:38.330215+0000) 2019-10-18T05:46:38.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:38.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:38.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:38.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:39.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:39.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:39.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:39.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:40.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:40.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:40.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:40.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:41.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:41.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:41.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:41.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:41.796 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:46:41.792+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:41.793577+0000) 2019-10-18T05:46:42.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:42.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:42.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:42.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:42.515 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:46:42.510+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:42.514496+0000) 2019-10-18T05:46:43.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:43.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:43.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:43.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:43.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:43.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:44.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:44.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:44.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:44.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:45.508 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:46:45.502+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:45.507959+0000) 2019-10-18T05:46:45.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:45.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:45.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:45.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:46.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:46.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:46.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:46.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:47.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:47.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:47.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:47.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:48.299 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:48.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:48.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:46:48.328+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:48.330322+0000) 2019-10-18T05:46:48.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:48.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:48.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:48.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:49.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:49.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:49.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:49.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:50.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:50.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:50.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:50.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:51.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:51.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:51.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:51.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:51.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:46:51.792+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:51.793674+0000) 2019-10-18T05:46:52.515 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:46:52.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:52.514688+0000) 2019-10-18T05:46:52.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:52.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:52.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:52.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:53.301 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:53.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:53.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:53.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:53.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:53.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:54.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:54.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:54.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:54.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:55.508 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:46:55.506+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:55.508166+0000) 2019-10-18T05:46:55.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:55.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:55.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:55.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:56.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:56.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:56.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:56.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:57.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:57.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:57.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:57.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:58.301 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:46:58.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:46:58.331 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:46:58.328+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:46:58.330429+0000) 2019-10-18T05:46:58.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:58.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:58.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:58.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:46:59.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:59.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:46:59.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:46:59.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:00.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:00.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:00.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:00.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:01.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:01.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:01.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:01.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:01.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:47:01.792+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:01.793772+0000) 2019-10-18T05:47:02.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:47:02.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:02.514900+0000) 2019-10-18T05:47:02.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:02.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:02.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:02.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:03.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:03.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:03.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:03.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:03.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:03.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:04.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:04.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:04.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:04.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:05.509 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:47:05.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:05.508358+0000) 2019-10-18T05:47:05.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:05.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:05.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:05.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:06.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:06.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:06.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:06.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:07.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:07.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:07.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:08.318 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:08.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:08.340 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:47:08.327+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:08.330537+0000) 2019-10-18T05:47:08.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:08.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:08.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:08.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:09.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:09.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:09.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:09.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:10.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:10.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:10.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:10.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:11.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:11.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:11.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:11.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:11.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:47:11.791+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:11.793874+0000) 2019-10-18T05:47:12.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:12.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:12.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:12.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:12.515 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:47:12.513+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:12.515090+0000) 2019-10-18T05:47:13.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:13.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:13.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:13.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:13.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:13.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:14.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:14.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:14.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:14.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:15.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:15.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:15.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:15.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:15.509 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:47:15.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:15.508566+0000) 2019-10-18T05:47:16.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:16.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:16.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:16.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:17.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:17.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:17.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:17.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:18.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:18.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:18.332 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:47:18.327+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:18.330644+0000) 2019-10-18T05:47:18.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:18.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:18.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:18.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:19.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:19.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:19.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:19.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:20.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:20.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:20.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:20.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:21.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:21.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:21.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:21.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:21.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:47:21.791+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:21.793971+0000) 2019-10-18T05:47:22.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:22.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:22.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:22.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:22.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:47:22.513+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:22.515282+0000) 2019-10-18T05:47:23.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:23.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:23.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:23.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:23.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:23.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:24.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:24.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:24.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:24.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:25.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:25.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:25.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:25.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:25.510 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:47:25.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:25.508714+0000) 2019-10-18T05:47:26.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:26.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:26.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:26.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:27.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:27.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:27.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:27.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:28.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:28.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:28.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:47:28.327+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:28.332133+0000) 2019-10-18T05:47:28.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:28.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:28.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:28.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:29.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:29.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:29.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:29.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:30.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:30.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:30.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:30.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:31.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:31.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:31.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:31.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:31.798 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:47:31.791+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:31.794068+0000) 2019-10-18T05:47:32.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:32.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:32.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:32.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:32.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:47:32.513+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:32.515461+0000) 2019-10-18T05:47:33.307 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:33.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:33.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:33.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:33.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:33.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:34.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:34.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:34.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:34.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:35.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:35.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:35.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:35.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:35.509 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:47:35.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:35.508882+0000) 2019-10-18T05:47:36.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:36.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:36.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:36.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:37.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:37.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:37.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:37.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:38.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:38.303+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:38.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:47:38.327+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:38.332263+0000) 2019-10-18T05:47:38.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:38.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:38.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:38.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:39.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:39.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:39.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:39.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:40.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:40.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:40.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:40.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:41.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:41.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:41.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:41.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:41.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:47:41.790+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:41.794165+0000) 2019-10-18T05:47:42.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:42.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:42.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:42.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:42.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:47:42.513+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:42.515646+0000) 2019-10-18T05:47:43.309 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:43.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:43.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:43.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:43.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:43.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:44.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:44.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:44.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:44.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:45.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:45.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:45.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:45.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:45.509 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:47:45.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:45.509087+0000) 2019-10-18T05:47:46.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:46.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:46.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:46.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:47.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:47.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:47.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:47.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:48.310 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:48.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:48.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:47:48.326+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:48.332367+0000) 2019-10-18T05:47:48.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:48.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:48.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:48.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:49.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:49.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:49.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:49.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:50.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:50.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:50.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:50.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:51.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:51.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:51.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:51.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:51.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:47:51.790+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:51.794260+0000) 2019-10-18T05:47:52.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:52.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:52.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:52.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:52.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:47:52.513+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:52.515859+0000) 2019-10-18T05:47:53.310 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:53.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:53.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:53.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:53.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:54.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:54.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:54.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:54.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:55.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:55.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:55.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:55.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:55.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:47:55.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:55.509300+0000) 2019-10-18T05:47:56.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:56.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:56.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:56.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:57.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:57.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:57.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:57.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:58.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:47:58.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:47:58.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:47:58.326+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:47:58.332471+0000) 2019-10-18T05:47:58.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:58.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:58.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:58.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:47:59.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:59.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:47:59.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:47:59.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:00.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:00.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:00.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:00.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:01.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:01.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:01.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:01.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:01.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:48:01.790+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:01.794366+0000) 2019-10-18T05:48:02.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:02.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:02.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:02.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:02.516 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:48:02.513+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:02.516068+0000) 2019-10-18T05:48:03.312 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:03.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:03.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:03.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:03.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:03.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:04.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:04.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:04.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:04.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:05.510 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:48:05.505+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:05.509488+0000) 2019-10-18T05:48:05.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:05.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:05.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:05.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:06.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:06.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:06.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:06.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:07.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:07.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:07.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:07.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:08.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:08.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:08.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:48:08.326+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:08.332574+0000) 2019-10-18T05:48:08.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:08.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:08.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:08.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:09.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:09.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:09.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:09.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:10.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:10.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:10.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:10.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:11.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:11.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:11.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:11.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:11.795 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:48:11.790+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:11.794464+0000) 2019-10-18T05:48:12.517 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:48:12.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:12.516250+0000) 2019-10-18T05:48:12.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:12.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:12.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:12.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:13.314 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:13.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:13.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:13.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:13.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:13.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:14.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:14.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:14.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:14.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:15.510 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:48:15.504+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:15.509670+0000) 2019-10-18T05:48:15.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:15.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:15.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:15.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:16.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:16.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:16.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:16.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:17.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:17.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:17.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:17.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:18.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:18.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:18.333 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:48:18.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:18.332695+0000) 2019-10-18T05:48:18.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:18.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:18.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:18.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:19.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:19.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:19.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:19.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:20.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:20.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:20.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:20.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:21.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:21.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:21.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:21.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:21.797 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:48:21.789+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:21.794559+0000) 2019-10-18T05:48:22.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:48:22.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:22.516444+0000) 2019-10-18T05:48:22.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:22.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:22.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:22.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:23.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:23.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:23.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:23.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:23.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:23.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:24.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:24.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:24.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:24.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:25.510 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:48:25.504+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:25.509860+0000) 2019-10-18T05:48:25.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:25.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:25.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:25.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:26.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:26.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:26.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:26.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:27.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:27.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:27.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:27.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:28.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:28.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:28.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:48:28.325+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:28.332800+0000) 2019-10-18T05:48:28.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:28.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:28.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:28.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:29.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:29.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:29.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:29.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:30.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:30.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:30.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:30.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:31.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:31.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:31.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:31.796 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:48:31.789+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:31.794659+0000) 2019-10-18T05:48:32.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:32.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:32.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:32.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:32.517 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:48:32.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:32.516637+0000) 2019-10-18T05:48:33.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:33.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:33.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:33.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:33.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:33.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:34.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:34.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:34.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:34.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:35.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:35.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:35.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:35.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:35.510 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:48:35.504+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:35.510042+0000) 2019-10-18T05:48:36.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:36.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:36.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:36.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:37.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:37.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:37.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:37.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:38.318 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:38.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:38.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:48:38.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:38.332916+0000) 2019-10-18T05:48:38.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:38.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:38.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:38.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:39.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:39.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:39.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:39.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:40.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:40.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:40.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:40.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:41.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:41.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:41.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:41.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:41.796 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:48:41.793+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:41.794763+0000) 2019-10-18T05:48:42.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:42.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:42.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:42.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:42.517 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:48:42.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:42.516825+0000) 2019-10-18T05:48:43.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:43.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:43.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:43.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:43.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:43.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:44.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:44.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:44.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:44.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:45.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:45.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:45.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:45.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:45.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:48:45.508+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:45.510199+0000) 2019-10-18T05:48:46.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:46.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:46.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:46.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:47.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:47.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:47.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:47.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:48.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:48.317+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:48.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:48:48.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:48.333038+0000) 2019-10-18T05:48:48.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:48.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:48.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:48.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:49.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:49.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:49.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:49.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:50.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:50.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:50.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:50.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:51.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:51.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:51.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:51.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:51.796 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:48:51.792+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:51.794859+0000) 2019-10-18T05:48:52.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:52.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:52.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:52.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:52.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:48:52.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:52.517031+0000) 2019-10-18T05:48:53.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:53.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:53.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:53.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:53.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:53.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:54.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:54.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:54.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:54.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:55.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:55.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:55.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:55.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:55.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:48:55.508+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:55.510332+0000) 2019-10-18T05:48:56.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:56.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:56.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:56.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:57.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:57.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:57.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:57.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:58.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:58.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:58.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:58.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:48:58.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:48:58.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:48:58.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:48:58.328+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:48:58.333149+0000) 2019-10-18T05:48:59.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:59.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:48:59.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:48:59.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:00.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:00.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:00.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:00.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:01.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:01.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:01.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:01.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:01.796 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:01.792+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:01.794955+0000) 2019-10-18T05:49:02.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:02.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:02.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:02.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:02.518 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:49:02.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:02.517247+0000) 2019-10-18T05:49:03.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:03.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:03.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:03.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:03.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:03.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:04.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:04.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:04.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:04.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:05.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:05.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:05.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:05.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:05.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:49:05.508+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:05.510511+0000) 2019-10-18T05:49:06.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:06.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:06.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:06.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:07.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:07.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:07.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:07.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:08.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:08.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:08.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:08.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:08.323 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:08.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:08.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:49:08.328+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:08.333257+0000) 2019-10-18T05:49:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:09.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:09.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:09.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:10.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:10.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:10.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:10.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:11.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:11.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:11.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:11.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:11.798 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:11.796+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:11.797463+0000) 2019-10-18T05:49:12.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:12.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:12.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:12.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:12.518 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:49:12.512+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:12.517460+0000) 2019-10-18T05:49:13.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:13.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:13.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:13.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:13.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:13.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:14.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:14.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:14.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:14.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:15.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:15.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:15.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:15.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:15.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:49:15.508+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:15.510673+0000) 2019-10-18T05:49:16.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:16.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:16.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:16.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:17.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:17.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:17.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:17.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:18.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:18.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:18.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:18.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:18.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:18.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:18.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:49:18.328+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:18.333363+0000) 2019-10-18T05:49:19.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:19.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:19.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:19.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:20.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:20.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:20.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:20.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:21.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:21.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:21.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:21.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:21.799 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:21.796+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:21.797551+0000) 2019-10-18T05:49:21.800 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:21.796+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:21.798867+0000) 2019-10-18T05:49:22.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:22.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:22.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:22.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:22.518 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:49:22.511+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:22.517644+0000) 2019-10-18T05:49:23.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:23.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:23.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:23.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:23.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:23.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:24.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:24.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:24.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:24.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:25.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:25.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:25.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:25.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:25.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:49:25.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:25.510842+0000) 2019-10-18T05:49:26.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:26.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:26.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:26.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:26.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:26.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:26.297748+0000) 2019-10-18T05:49:27.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:27.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:27.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:27.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:28.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:28.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:28.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:28.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:28.327 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:28.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:28.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:49:28.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:28.333470+0000) 2019-10-18T05:49:29.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:29.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:29.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:29.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:30.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:30.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:30.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:30.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:31.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:31.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:31.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:31.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:32.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:32.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:32.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:32.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:32.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:49:32.511+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:32.517852+0000) 2019-10-18T05:49:33.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:33.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:33.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:33.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:33.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:33.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:34.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:34.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:34.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:34.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:35.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:35.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:35.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:35.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:35.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:49:35.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:35.510997+0000) 2019-10-18T05:49:36.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:36.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:36.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:36.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:36.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:36.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:36.297852+0000) 2019-10-18T05:49:37.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:37.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:37.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:37.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:38.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:38.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:38.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:38.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:38.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:38.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:38.334 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:49:38.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:38.333577+0000) 2019-10-18T05:49:39.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:39.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:39.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:39.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:40.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:40.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:40.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:40.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:41.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:41.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:41.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:41.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:42.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:42.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:42.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:42.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:42.518 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:49:42.511+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:42.518024+0000) 2019-10-18T05:49:43.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:43.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:43.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:43.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:43.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:43.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:44.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:44.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:44.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:44.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:45.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:45.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:45.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:45.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:45.511 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:49:45.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:45.511176+0000) 2019-10-18T05:49:46.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:46.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:46.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:46.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:46.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:46.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:46.297952+0000) 2019-10-18T05:49:47.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:47.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:47.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:47.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:48.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:48.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:48.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:48.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:48.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:48.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:48.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:49:48.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:48.333690+0000) 2019-10-18T05:49:49.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:49.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:49.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:49.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:50.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:50.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:50.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:50.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:51.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:51.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:51.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:51.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:52.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:52.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:52.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:52.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:52.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:49:52.515+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:52.518228+0000) 2019-10-18T05:49:53.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:53.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:53.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:53.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:53.331 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:53.327+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:54.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:54.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:54.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:54.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:55.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:55.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:55.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:55.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:55.512 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:49:55.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:55.511316+0000) 2019-10-18T05:49:56.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:56.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:56.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:56.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:56.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:49:56.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:56.298049+0000) 2019-10-18T05:49:57.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:57.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:57.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:57.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:58.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:58.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:58.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:58.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:49:58.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:49:58.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:49:58.335 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:49:58.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:49:58.333809+0000) 2019-10-18T05:49:59.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:59.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:49:59.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:49:59.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:00.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:00.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:00.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:00.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:01.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:01.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:01.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:01.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:02.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:02.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:02.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:02.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:02.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:50:02.515+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:02.518417+0000) 2019-10-18T05:50:03.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:03.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:03.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:03.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:03.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:03.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:04.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:04.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:04.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:04.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:05.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:05.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:05.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:05.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:05.512 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:50:05.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:05.511499+0000) 2019-10-18T05:50:06.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:06.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:06.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:06.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:06.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:50:06.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:06.298149+0000) 2019-10-18T05:50:07.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:07.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:07.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:07.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:08.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:08.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:39:35.590100+0000 (oldest deadline 2019-10-18T05:39:55.590100+0000) 2019-10-18T05:50:08.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:08.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:08.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:08.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:08.335 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:50:08.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:08.333919+0000) 2019-10-18T05:50:09.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:09.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:10.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:10.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:11.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:11.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:12.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:12.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:12.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:50:12.515+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:12.518555+0000) 2019-10-18T05:50:13.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:13.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:13.334 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:13.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:14.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:14.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:15.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:15.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:15.512 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:50:15.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:15.511678+0000) 2019-10-18T05:50:16.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:16.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:16.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:50:16.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:16.298247+0000) 2019-10-18T05:50:17.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:17.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:18.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:18.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:18.335 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:50:18.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:18.334046+0000) 2019-10-18T05:50:18.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:18.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:19.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:19.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:20.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:20.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:21.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:21.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:22.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:22.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:22.524 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:50:22.515+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:22.518760+0000) 2019-10-18T05:50:23.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:23.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:23.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:23.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:24.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:24.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:25.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:25.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:25.512 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:50:25.507+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:25.511815+0000) 2019-10-18T05:50:26.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:26.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:26.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:50:26.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:26.298347+0000) 2019-10-18T05:50:27.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:27.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:28.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:28.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:28.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:50:28.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:28.334154+0000) 2019-10-18T05:50:28.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:28.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:29.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:29.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:29.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:29.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:30.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:30.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:30.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:30.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:31.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:31.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:31.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:31.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:32.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:32.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:32.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:32.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:32.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:50:32.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:32.518939+0000) 2019-10-18T05:50:33.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:33.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:33.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:33.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:33.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:33.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:34.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:34.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:34.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:34.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:35.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:35.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:35.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:35.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:35.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:50:35.506+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:35.511961+0000) 2019-10-18T05:50:36.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:36.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:36.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:36.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:36.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:50:36.293+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:36.298447+0000) 2019-10-18T05:50:37.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:37.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:37.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:37.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:38.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:38.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:38.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:38.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:38.335 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:50:38.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:38.334260+0000) 2019-10-18T05:50:38.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:38.333+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:39.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:39.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:39.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:39.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:40.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:40.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:40.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:40.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:41.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:41.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:41.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:41.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:42.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:42.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:42.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:42.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:42.519 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:50:42.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:42.519078+0000) 2019-10-18T05:50:43.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:43.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:43.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:43.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:43.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:43.333+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:44.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:44.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:44.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:44.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:45.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:45.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:45.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:45.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:45.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:50:45.506+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:45.512124+0000) 2019-10-18T05:50:46.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:46.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:46.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:46.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:46.299 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:50:46.293+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:46.298542+0000) 2019-10-18T05:50:47.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:47.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:47.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:47.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:48.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:48.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:48.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:48.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:48.335 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:50:48.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:48.334362+0000) 2019-10-18T05:50:48.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:48.333+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:49.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:49.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:49.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:49.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:50.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:50.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:50.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:50.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:51.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:51.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:51.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:51.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:52.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:52.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:52.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:52.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:52.520 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:50:52.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:52.519257+0000) 2019-10-18T05:50:53.342 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:53.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:53.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:53.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:53.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:53.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:54.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:54.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:54.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:54.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:55.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:55.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:55.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:55.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:55.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:50:55.506+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:55.512287+0000) 2019-10-18T05:50:56.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:50:56.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:56.298640+0000) 2019-10-18T05:50:56.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:56.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:56.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:56.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:57.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:57.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:57.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:57.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:58.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:50:58.333+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:50:58.334468+0000) 2019-10-18T05:50:58.343 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:50:58.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:50:58.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:58.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:58.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:58.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:50:59.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:59.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:50:59.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:50:59.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:00.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:00.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:00.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:00.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:01.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:01.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:01.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:01.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:02.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:02.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:02.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:02.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:02.520 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:51:02.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:02.519472+0000) 2019-10-18T05:51:03.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:03.341+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:03.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:03.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:03.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:03.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:04.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:04.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:04.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:04.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:05.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:05.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:05.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:05.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:05.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:51:05.506+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:05.512455+0000) 2019-10-18T05:51:06.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:51:06.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:06.298736+0000) 2019-10-18T05:51:06.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:06.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:06.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:06.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:07.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:07.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:07.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:07.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:08.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:51:08.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:08.334574+0000) 2019-10-18T05:51:08.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:08.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:08.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:08.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:08.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:08.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:09.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:09.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:09.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:09.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:10.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:10.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:10.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:10.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:11.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:11.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:12.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:12.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:12.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:12.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:12.520 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:51:12.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:12.519616+0000) 2019-10-18T05:51:13.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:13.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:13.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:13.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:13.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:13.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:14.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:14.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:14.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:14.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:15.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:15.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:15.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:15.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:15.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:51:15.506+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:15.512599+0000) 2019-10-18T05:51:16.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:51:16.296+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:16.298831+0000) 2019-10-18T05:51:16.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:16.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:16.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:16.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:17.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:17.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:17.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:17.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:18.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:18.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:18.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:18.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:18.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:51:18.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:18.334679+0000) 2019-10-18T05:51:18.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:18.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:19.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:19.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:19.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:19.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:20.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:20.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:20.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:20.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:21.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:21.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:21.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:21.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:22.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:22.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:22.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:22.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:22.520 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:51:22.514+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:22.519810+0000) 2019-10-18T05:51:23.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:23.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:23.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:23.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:23.347 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:23.344+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:24.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:24.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:24.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:24.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:25.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:25.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:25.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:25.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:25.513 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:51:25.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:25.512787+0000) 2019-10-18T05:51:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:26.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:26.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:26.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:26.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:51:26.296+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:26.298929+0000) 2019-10-18T05:51:27.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:27.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:27.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:27.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:28.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:28.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:28.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:28.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:28.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:51:28.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:28.334787+0000) 2019-10-18T05:51:28.347 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:28.344+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:29.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:29.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:29.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:29.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:30.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:30.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:30.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:30.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:31.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:31.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:31.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:31.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:32.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:32.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:32.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:32.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:32.520 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:51:32.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:32.519993+0000) 2019-10-18T05:51:33.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:33.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:33.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:33.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:33.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:33.344+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:34.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:34.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:34.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:34.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:35.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:35.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:35.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:35.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:35.514 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:51:35.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:35.512976+0000) 2019-10-18T05:51:36.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:36.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:36.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:36.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:36.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:51:36.296+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:36.299026+0000) 2019-10-18T05:51:37.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:37.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:37.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:37.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:38.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:38.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:38.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:38.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:38.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:51:38.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:38.334888+0000) 2019-10-18T05:51:38.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:38.344+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:39.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:39.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:39.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:39.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:40.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:40.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:40.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:40.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:41.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:41.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:41.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:41.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:42.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:42.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:42.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:42.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:42.521 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:51:42.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:42.520174+0000) 2019-10-18T05:51:43.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:43.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:43.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:43.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:43.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:43.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:44.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:44.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:44.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:44.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:45.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:45.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:45.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:45.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:45.514 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:51:45.509+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:45.513154+0000) 2019-10-18T05:51:46.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:46.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:46.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:46.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:46.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:51:46.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:46.299125+0000) 2019-10-18T05:51:47.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:47.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:47.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:47.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:48.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:48.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:48.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:48.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:48.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:51:48.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:48.334991+0000) 2019-10-18T05:51:48.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:48.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:49.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:49.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:49.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:49.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:50.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:50.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:50.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:50.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:51.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:51.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:51.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:51.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:52.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:52.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:52.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:52.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:52.521 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:51:52.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:52.520356+0000) 2019-10-18T05:51:53.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:53.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:53.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:53.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:53.352 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:53.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:54.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:54.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:54.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:54.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:55.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:55.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:55.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:55.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:55.514 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:51:55.509+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:55.513297+0000) 2019-10-18T05:51:56.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:56.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:56.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:56.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:56.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:51:56.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:56.299225+0000) 2019-10-18T05:51:57.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:57.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:57.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:57.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:58.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:58.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:58.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:51:58.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:51:58.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:51:58.335095+0000) 2019-10-18T05:51:58.353 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:51:58.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:51:59.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:59.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:51:59.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:51:59.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:00.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:00.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:00.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:00.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:01.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:01.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:01.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:01.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:02.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:02.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:02.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:02.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:02.521 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:52:02.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:02.520567+0000) 2019-10-18T05:52:03.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:03.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:03.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:03.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:03.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:03.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:04.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:04.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:04.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:04.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:05.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:05.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:05.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:05.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:05.514 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:52:05.509+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:05.513476+0000) 2019-10-18T05:52:06.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:06.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:06.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:06.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:06.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:52:06.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:06.299326+0000) 2019-10-18T05:52:07.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:07.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:07.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:08.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:08.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:08.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:08.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:08.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:52:08.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:08.335224+0000) 2019-10-18T05:52:08.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:08.351+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:09.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:09.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:09.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:10.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:10.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:10.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:10.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:11.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:11.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:11.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:11.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:12.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:12.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:12.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:12.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:12.524 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:52:12.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:12.520760+0000) 2019-10-18T05:52:13.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:13.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:13.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:13.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:13.356 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:13.351+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:14.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:14.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:14.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:14.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:15.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:15.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:15.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:15.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:15.514 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:52:15.509+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:15.513684+0000) 2019-10-18T05:52:16.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:16.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:16.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:16.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:16.300 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:52:16.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:16.299424+0000) 2019-10-18T05:52:17.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:17.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:17.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:17.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:18.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:18.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:18.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:18.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:18.336 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:52:18.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:18.335347+0000) 2019-10-18T05:52:18.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:18.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:19.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:19.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:19.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:19.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:19.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:19.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:19.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:19.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:20.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:20.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:20.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:20.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:21.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:21.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:21.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:21.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:22.521 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:52:22.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:22.520935+0000) 2019-10-18T05:52:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:23.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:23.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:23.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:23.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:24.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:24.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:24.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:24.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:25.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:25.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:25.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:25.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:25.515 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:52:25.509+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:25.513855+0000) 2019-10-18T05:52:26.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:26.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:26.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:26.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:26.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:52:26.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:26.299523+0000) 2019-10-18T05:52:27.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:27.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:27.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:27.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:28.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:28.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:28.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:28.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:28.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:52:28.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:28.335467+0000) 2019-10-18T05:52:28.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:28.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:29.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:29.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:29.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:29.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:30.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:30.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:30.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:30.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:31.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:31.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:31.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:31.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:32.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:32.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:32.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:32.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:32.522 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:52:32.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:32.521155+0000) 2019-10-18T05:52:33.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:33.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:33.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:33.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:33.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:33.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:34.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:34.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:34.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:34.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:35.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:35.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:35.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:35.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:35.514 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:52:35.509+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:35.514033+0000) 2019-10-18T05:52:36.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:36.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:36.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:36.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:36.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:52:36.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:36.299622+0000) 2019-10-18T05:52:37.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:37.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:37.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:37.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:38.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:38.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:38.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:38.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:38.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:52:38.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:38.335583+0000) 2019-10-18T05:52:38.360 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:38.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:39.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:39.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:39.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:39.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:40.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:40.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:40.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:40.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:41.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:41.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:41.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:41.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:42.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:42.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:42.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:42.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:42.522 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:52:42.517+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:42.521334+0000) 2019-10-18T05:52:43.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:43.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:43.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:43.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:43.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:43.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:44.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:44.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:44.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:44.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:45.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:45.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:45.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:45.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:45.515 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:52:45.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:45.514175+0000) 2019-10-18T05:52:46.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:52:46.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:46.299720+0000) 2019-10-18T05:52:46.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:46.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:46.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:46.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:47.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:47.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:47.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:47.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:48.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:52:48.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:48.335707+0000) 2019-10-18T05:52:48.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:48.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:48.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:48.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:48.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:48.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:49.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:49.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:49.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:49.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:50.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:50.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:50.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:50.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:51.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:51.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:51.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:51.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:52.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:52.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:52.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:52.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:52.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:52:52.516+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:52.521521+0000) 2019-10-18T05:52:53.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:53.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:53.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:53.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:53.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:53.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:54.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:54.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:54.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:54.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:55.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:55.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:55.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:55.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:55.515 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:52:55.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:55.514368+0000) 2019-10-18T05:52:56.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:52:56.293+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:56.299819+0000) 2019-10-18T05:52:56.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:56.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:56.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:56.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:57.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:57.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:57.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:57.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:58.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:58.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:58.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:58.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:52:58.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:52:58.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:52:58.335831+0000) 2019-10-18T05:52:58.376 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:52:58.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:52:59.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:59.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:52:59.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:52:59.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:00.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:00.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:00.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:00.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:01.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:01.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:01.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:01.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:02.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:02.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:02.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:02.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:02.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:53:02.516+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:02.521700+0000) 2019-10-18T05:53:03.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:03.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:03.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:03.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:03.364 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:03.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:04.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:04.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:04.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:04.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:05.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:05.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:05.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:05.515 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:53:05.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:05.514559+0000) 2019-10-18T05:53:06.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:06.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:06.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:06.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:06.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:53:06.293+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:06.299924+0000) 2019-10-18T05:53:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:07.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:07.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:08.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:08.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:08.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:08.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:08.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:53:08.329+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:08.335944+0000) 2019-10-18T05:53:08.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:08.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:09.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:09.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:09.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:09.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:10.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:10.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:10.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:10.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:11.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:11.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:11.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:11.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:12.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:12.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:12.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:12.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:12.522 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:53:12.516+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:12.521880+0000) 2019-10-18T05:53:13.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:13.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:13.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:13.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:13.366 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:13.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:14.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:14.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:14.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:14.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:15.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:15.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:15.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:15.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:15.515 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:53:15.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:15.514739+0000) 2019-10-18T05:53:16.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:16.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:16.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:16.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:16.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:53:16.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:16.300021+0000) 2019-10-18T05:53:17.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:17.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:17.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:17.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:18.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:18.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:18.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:18.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:18.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:53:18.333+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:18.336068+0000) 2019-10-18T05:53:18.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:18.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:19.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:19.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:19.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:19.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:20.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:20.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:20.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:20.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:21.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:21.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:21.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:21.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:22.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:22.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:22.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:22.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:22.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:53:22.516+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:22.522062+0000) 2019-10-18T05:53:23.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:23.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:23.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:23.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:23.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:23.365+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:24.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:24.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:24.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:24.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:25.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:25.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:25.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:25.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:25.515 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:53:25.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:25.514944+0000) 2019-10-18T05:53:26.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:26.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:26.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:26.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:26.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:53:26.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:26.300117+0000) 2019-10-18T05:53:27.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:27.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:27.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:27.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:28.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:28.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:28.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:28.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:28.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:53:28.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:28.336204+0000) 2019-10-18T05:53:28.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:28.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:29.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:29.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:29.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:29.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:30.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:30.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:30.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:30.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:31.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:31.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:31.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:31.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:32.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:32.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:32.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:32.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:32.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:53:32.520+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:32.522238+0000) 2019-10-18T05:53:33.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:33.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:33.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:33.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:33.370 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:33.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:34.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:34.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:34.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:34.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:35.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:35.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:35.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:35.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:35.516 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:53:35.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:35.515129+0000) 2019-10-18T05:53:36.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:36.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:36.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:36.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:36.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:53:36.296+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:36.300214+0000) 2019-10-18T05:53:37.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:37.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:37.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:37.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:38.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:38.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:38.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:38.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:38.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:53:38.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:38.336318+0000) 2019-10-18T05:53:38.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:38.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:39.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:39.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:39.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:39.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:40.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:40.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:40.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:40.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:41.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:41.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:41.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:41.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:42.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:42.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:42.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:42.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:42.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:53:42.520+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:42.522414+0000) 2019-10-18T05:53:43.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:43.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:43.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:43.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:43.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:43.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:44.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:44.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:44.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:44.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:45.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:45.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:45.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:45.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:45.516 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:53:45.512+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:45.515288+0000) 2019-10-18T05:53:46.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:46.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:46.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:46.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:46.301 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:53:46.296+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:46.300311+0000) 2019-10-18T05:53:47.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:47.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:47.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:47.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:48.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:48.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:48.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:48.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:48.337 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:53:48.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:48.336442+0000) 2019-10-18T05:53:48.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:48.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:49.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:49.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:49.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:49.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:50.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:50.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:50.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:50.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:51.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:51.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:51.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:51.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:52.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:52.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:52.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:52.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:52.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:53:52.520+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:52.522603+0000) 2019-10-18T05:53:53.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:53.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:53.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:53.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:53.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:53.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:54.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:54.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:54.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:54.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:55.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:55.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:55.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:55.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:55.516 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:53:55.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:55.515467+0000) 2019-10-18T05:53:56.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:56.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:56.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:56.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:56.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:53:56.296+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:56.300410+0000) 2019-10-18T05:53:57.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:57.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:57.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:57.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:58.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:58.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:58.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:58.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:53:58.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:53:58.332+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:53:58.336572+0000) 2019-10-18T05:53:58.374 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:53:58.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:53:59.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:59.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:53:59.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:53:59.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:00.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:00.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:00.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:00.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:01.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:01.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:01.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:01.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:02.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:02.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:02.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:02.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:02.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:54:02.519+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:02.522786+0000) 2019-10-18T05:54:03.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:03.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:03.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:03.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:03.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:03.367+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:04.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:04.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:04.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:04.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:05.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:05.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:05.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:05.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:05.516 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:54:05.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:05.515656+0000) 2019-10-18T05:54:06.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:06.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:06.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:06.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:06.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:54:06.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:06.300511+0000) 2019-10-18T05:54:07.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:07.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:07.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:07.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:08.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:08.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:08.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:08.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:08.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:54:08.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:08.336690+0000) 2019-10-18T05:54:08.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:08.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:09.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:09.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:09.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:09.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:10.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:10.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:10.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:10.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:11.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:11.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:11.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:11.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:12.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:12.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:12.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:12.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:12.523 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:54:12.519+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:12.522972+0000) 2019-10-18T05:54:13.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:13.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:13.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:13.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:13.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:13.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:14.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:14.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:14.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:14.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:15.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:15.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:15.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:15.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:15.516 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:54:15.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:15.515834+0000) 2019-10-18T05:54:16.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:16.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:16.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:16.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:16.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:54:16.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:16.300606+0000) 2019-10-18T05:54:17.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:17.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:17.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:17.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:18.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:18.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:18.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:18.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:18.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:54:18.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:18.336812+0000) 2019-10-18T05:54:18.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:18.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:19.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:19.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:19.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:19.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:20.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:20.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:20.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:20.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:21.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:21.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:21.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:21.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:22.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:22.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:22.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:22.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:22.524 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:54:22.519+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:22.523153+0000) 2019-10-18T05:54:23.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:23.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:23.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:23.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:23.378 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:23.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:24.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:24.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:24.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:24.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:25.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:25.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:25.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:25.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:25.517 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:54:25.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:25.516010+0000) 2019-10-18T05:54:26.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:25.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:26.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:25.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:26.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:54:26.295+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:26.300703+0000) 2019-10-18T05:54:26.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:26.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:26.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:26.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:28.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:28.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:28.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:28.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:28.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:54:28.331+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:28.336933+0000) 2019-10-18T05:54:28.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:28.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:29.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:29.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:29.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:29.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:30.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:30.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:30.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:30.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:31.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:31.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:31.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:31.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:32.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:32.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:32.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:32.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:32.524 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:54:32.519+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:32.523333+0000) 2019-10-18T05:54:33.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:33.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:33.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:33.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:33.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:33.375+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:34.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:34.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:34.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:34.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:35.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:35.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:35.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:35.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:35.517 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:54:35.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:35.516198+0000) 2019-10-18T05:54:36.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:36.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:36.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:36.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:36.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:54:36.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:36.300802+0000) 2019-10-18T05:54:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:37.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:37.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:37.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:38.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:38.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:38.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:38.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:38.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:54:38.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:38.337051+0000) 2019-10-18T05:54:38.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:38.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:39.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:39.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:39.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:39.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:40.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:40.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:40.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:40.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:41.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:41.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:41.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:41.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:42.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:42.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:42.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:42.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:42.524 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:54:42.519+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:42.523523+0000) 2019-10-18T05:54:43.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:43.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:43.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:43.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:43.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:43.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:44.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:44.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:44.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:44.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:45.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:45.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:45.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:45.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:45.517 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:54:45.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:45.516383+0000) 2019-10-18T05:54:46.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:46.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:46.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:46.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:46.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:54:46.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:46.300901+0000) 2019-10-18T05:54:47.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:47.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:47.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:47.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:48.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:48.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:48.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:48.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:48.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:54:48.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:48.337175+0000) 2019-10-18T05:54:48.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:48.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:49.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:49.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:49.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:49.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:50.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:50.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:50.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:50.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:51.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:51.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:51.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:51.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:52.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:52.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:52.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:52.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:52.524 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:54:52.519+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:52.523713+0000) 2019-10-18T05:54:53.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:53.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:53.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:53.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:53.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:53.378+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:54.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:54.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:54.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:54.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:55.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:55.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:55.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:55.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:55.517 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:54:55.511+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:55.516543+0000) 2019-10-18T05:54:56.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:56.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:56.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:56.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:56.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:54:56.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:56.301004+0000) 2019-10-18T05:54:56.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:56.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:56.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:56.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:58.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:58.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:58.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:54:58.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:54:58.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:54:58.337298+0000) 2019-10-18T05:54:58.386 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:54:58.378+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:54:59.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:59.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:54:59.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:54:59.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:00.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:00.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:00.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:00.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:01.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:01.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:01.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:01.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:02.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:02.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:02.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:02.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:02.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:55:02.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:02.523898+0000) 2019-10-18T05:55:03.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:03.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:03.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:03.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:03.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:03.378+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:04.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:04.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:04.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:04.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:05.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:05.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:05.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:05.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:05.517 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:55:05.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:05.516731+0000) 2019-10-18T05:55:06.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:06.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:06.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:06.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:06.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:55:06.294+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:06.301110+0000) 2019-10-18T05:55:07.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:07.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:07.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:07.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:08.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:08.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:08.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:08.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:08.338 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:55:08.330+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:08.337413+0000) 2019-10-18T05:55:08.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:08.378+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:09.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:09.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:09.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:09.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:10.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:10.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:10.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:10.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:11.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:11.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:11.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:11.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:12.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:12.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:12.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:12.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:12.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:55:12.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:12.524102+0000) 2019-10-18T05:55:13.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:13.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:13.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:13.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:13.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:13.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:14.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:14.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:14.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:14.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:15.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:15.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:15.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:15.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:15.518 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:55:15.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:15.516914+0000) 2019-10-18T05:55:16.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:16.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:16.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:16.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:16.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:55:16.293+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:16.301206+0000) 2019-10-18T05:55:17.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:17.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:17.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:17.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:18.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:18.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:18.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:18.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:55:18.333+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:18.337500+0000) 2019-10-18T05:55:18.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:18.381+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:19.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:19.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:19.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:19.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:20.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:20.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:20.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:20.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:21.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:21.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:21.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:21.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:22.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:22.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:22.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:22.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:22.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:55:22.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:22.524288+0000) 2019-10-18T05:55:23.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:23.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:23.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:23.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:23.388 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:23.381+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:24.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:24.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:24.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:24.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:25.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:25.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:25.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:25.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:25.518 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:55:25.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:25.517096+0000) 2019-10-18T05:55:26.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:26.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:26.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:26.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:26.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:55:26.293+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:26.301306+0000) 2019-10-18T05:55:27.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:27.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:27.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:27.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:28.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:28.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:28.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:28.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:28.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:55:28.333+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:28.337618+0000) 2019-10-18T05:55:28.389 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:28.381+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:29.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:29.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:29.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:29.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:30.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:30.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:30.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:30.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:31.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:31.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:31.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:31.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:32.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:32.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:32.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:32.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:32.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:55:32.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:32.524463+0000) 2019-10-18T05:55:33.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:33.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:33.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:33.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:33.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:33.385+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:34.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:34.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:34.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:34.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:35.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:35.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:35.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:35.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:35.518 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:55:35.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:35.517270+0000) 2019-10-18T05:55:36.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:36.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:36.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:36.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:36.302 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:55:36.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:36.301408+0000) 2019-10-18T05:55:37.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:37.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:37.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:37.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:38.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:38.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:38.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:38.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:38.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:55:38.337+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:38.337742+0000) 2019-10-18T05:55:38.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:38.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:39.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:39.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:39.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:39.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:40.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:40.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:40.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:40.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:41.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:41.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:41.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:41.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:42.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:42.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:42.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:42.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:42.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:55:42.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:42.524644+0000) 2019-10-18T05:55:43.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:43.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:43.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:43.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:43.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:43.389+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:44.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:44.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:44.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:44.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:45.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:45.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:45.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:45.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:45.518 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:55:45.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:45.517456+0000) 2019-10-18T05:55:45.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:45.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:45.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:45.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:46.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:55:46.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:46.301463+0000) 2019-10-18T05:55:47.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:46.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:47.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:46.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:47.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:47.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:47.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:47.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:48.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:55:48.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:48.337878+0000) 2019-10-18T05:55:48.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:48.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:48.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:48.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:48.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:48.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:49.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:49.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:49.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:49.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:51.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:51.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:51.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:51.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:52.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:52.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:52.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:52.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:52.525 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:55:52.518+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:52.524822+0000) 2019-10-18T05:55:53.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:53.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:53.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:53.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:53.393 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:53.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:54.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:54.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:54.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:54.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:55.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:55.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:55.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:55.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:55:55.510+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:55.517636+0000) 2019-10-18T05:55:56.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:56.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:56.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:56.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:56.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:55:56.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:56.301565+0000) 2019-10-18T05:55:57.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:57.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:57.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:57.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:58.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:58.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:58.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:58.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:55:58.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:55:58.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:55:58.337994+0000) 2019-10-18T05:55:58.394 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:55:58.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:55:59.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:59.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:55:59.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:55:59.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:00.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:00.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:00.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:00.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:01.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:01.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:01.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:01.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:02.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:02.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:02.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:02.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:02.526 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:56:02.522+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:02.525002+0000) 2019-10-18T05:56:03.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:03.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:03.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:03.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:03.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:03.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:04.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:04.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:04.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:04.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:05.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:05.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:05.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:05.518 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:56:05.514+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:05.517816+0000) 2019-10-18T05:56:06.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:06.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:06.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:06.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:06.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:56:06.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:06.301666+0000) 2019-10-18T05:56:07.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:07.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:07.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:07.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:08.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:08.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:08.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:08.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:08.340 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:56:08.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:08.338106+0000) 2019-10-18T05:56:08.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:08.392+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:09.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:09.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:09.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:09.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:10.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:10.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:10.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:10.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:11.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:11.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:11.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:11.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:12.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:12.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:12.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:12.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:12.526 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:56:12.521+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:12.525214+0000) 2019-10-18T05:56:13.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:13.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:13.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:13.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:13.397 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:13.392+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:14.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:14.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:14.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:14.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:15.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:15.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:15.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:15.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:15.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:56:15.513+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:15.517989+0000) 2019-10-18T05:56:16.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:16.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:16.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:16.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:16.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:56:16.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:16.301762+0000) 2019-10-18T05:56:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:17.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:17.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:17.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:18.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:18.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:18.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:18.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:18.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:56:18.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:18.338229+0000) 2019-10-18T05:56:18.397 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:18.392+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:19.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:19.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:19.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:19.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:20.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:20.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:20.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:20.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:21.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:21.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:21.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:21.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:22.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:22.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:22.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:22.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:22.526 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:56:22.521+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:22.525410+0000) 2019-10-18T05:56:23.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:23.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:23.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:23.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:23.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:23.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:24.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:24.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:24.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:25.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:25.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:25.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:25.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:25.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:56:25.517+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:25.518169+0000) 2019-10-18T05:56:26.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:26.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:26.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:26.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:26.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:56:26.299+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:26.301864+0000) 2019-10-18T05:56:27.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:27.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:27.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:27.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:28.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:28.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:28.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:28.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:28.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:56:28.335+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:28.338349+0000) 2019-10-18T05:56:28.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:28.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:29.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:29.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:29.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:29.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:30.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:30.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:30.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:30.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:31.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:31.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:31.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:31.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:32.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:32.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:32.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:32.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:32.526 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:56:32.521+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:32.525590+0000) 2019-10-18T05:56:33.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:33.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:33.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:33.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:33.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:33.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:34.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:34.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:34.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:34.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:35.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:35.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:35.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:35.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:35.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:56:35.517+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:35.518380+0000) 2019-10-18T05:56:36.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:36.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:36.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:36.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:36.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:56:36.299+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:36.301967+0000) 2019-10-18T05:56:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:37.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:37.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:38.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:38.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:38.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:38.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:38.339 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:56:38.335+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:38.338463+0000) 2019-10-18T05:56:38.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:38.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:39.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:39.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:39.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:39.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:40.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:40.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:40.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:40.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:41.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:41.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:41.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:41.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:42.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:42.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:42.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:42.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:42.526 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:56:42.521+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:42.525804+0000) 2019-10-18T05:56:43.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:43.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:43.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:43.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:43.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:43.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:44.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:44.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:44.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:44.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:45.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:45.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:45.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:45.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:45.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:56:45.517+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:45.518564+0000) 2019-10-18T05:56:46.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:46.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:46.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:46.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:46.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:56:46.299+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:46.302066+0000) 2019-10-18T05:56:47.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:47.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:47.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:47.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:48.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:48.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:48.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:48.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:48.340 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:56:48.335+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:48.338585+0000) 2019-10-18T05:56:48.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:48.399+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:49.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:49.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:49.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:49.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:50.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:50.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:50.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:50.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:51.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:51.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:51.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:51.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:52.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:52.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:52.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:52.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:52.526 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:56:52.521+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:52.525928+0000) 2019-10-18T05:56:53.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:53.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:53.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:53.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:53.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:53.399+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:54.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:54.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:54.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:54.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 480 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:55.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:55.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:55.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:55.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:55.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:56:55.517+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:55.518696+0000) 2019-10-18T05:56:56.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:56.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:56.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:56.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:56.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:56:56.298+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:56.302171+0000) 2019-10-18T05:56:57.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:57.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:57.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:57.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:58.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:58.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:58.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:58.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:56:58.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:56:58.334+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:56:58.338703+0000) 2019-10-18T05:56:58.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:56:58.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:56:59.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:59.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:56:59.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:56:59.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:00.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:00.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:00.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:00.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:01.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:01.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:01.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:01.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:02.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:02.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:02.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:02.527 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:57:02.521+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:02.526040+0000) 2019-10-18T05:57:03.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:03.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:03.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:03.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:03.405 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:03.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:04.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:04.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:04.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:04.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:05.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:05.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:05.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:05.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:05.519 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:05.517+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:05.518812+0000) 2019-10-18T05:57:06.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:06.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:06.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:06.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:06.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:57:06.298+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:06.302273+0000) 2019-10-18T05:57:07.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:07.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:07.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:07.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:08.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:08.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:08.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:08.340 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:57:08.334+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:08.338827+0000) 2019-10-18T05:57:08.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:08.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:09.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:09.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:09.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:09.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:10.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:10.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:10.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:10.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:11.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:11.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:11.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:11.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:12.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:12.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:12.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:12.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:12.527 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:57:12.525+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:12.526159+0000) 2019-10-18T05:57:12.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:12.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:12.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:12.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:13.407 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:13.402+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:14.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:14.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:14.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:14.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:15.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:15.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:15.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:15.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:15.521 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:15.517+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:15.518925+0000) 2019-10-18T05:57:16.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:16.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:16.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:16.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:16.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:57:16.298+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:16.302371+0000) 2019-10-18T05:57:17.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:17.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:17.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:17.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:18.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:18.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:18.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:18.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:18.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:57:18.334+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:18.338952+0000) 2019-10-18T05:57:18.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:18.402+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:19.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:19.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:19.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:19.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:20.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:20.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:20.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:20.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:21.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:21.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:21.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:21.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:22.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:22.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:22.527 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:57:22.524+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:22.526294+0000) 2019-10-18T05:57:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:23.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:23.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:23.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:23.402+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:24.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:24.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:25.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:25.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:25.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:25.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:25.521 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:25.516+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:25.519033+0000) 2019-10-18T05:57:26.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:26.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:26.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:26.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:26.303 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:57:26.298+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:26.302470+0000) 2019-10-18T05:57:27.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:27.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:27.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:27.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:28.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:28.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:28.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:28.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:28.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:57:28.334+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:28.339073+0000) 2019-10-18T05:57:28.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:28.402+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:29.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:29.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:29.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:29.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:30.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:30.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:30.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:30.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:31.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:31.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:31.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:31.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:32.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:32.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:32.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:32.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:32.527 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:57:32.524+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:32.526400+0000) 2019-10-18T05:57:33.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:33.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:33.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:33.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:33.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:33.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:34.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:34.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:34.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:34.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:35.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:35.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:35.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:35.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:35.520 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:35.516+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:35.519143+0000) 2019-10-18T05:57:35.521 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:35.516+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:35.520313+0000) 2019-10-18T05:57:36.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:57:36.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:36.302570+0000) 2019-10-18T05:57:36.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:36.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:36.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:36.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:37.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:37.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:37.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:37.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:38.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:38.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:38.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:38.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:38.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:57:38.333+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:38.339188+0000) 2019-10-18T05:57:38.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:38.405+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:39.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:39.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:39.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:39.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:40.020 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:40.016+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:40.019383+0000) 2019-10-18T05:57:40.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:40.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:40.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:40.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:41.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:41.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:41.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:41.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:42.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:42.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:42.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:42.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:42.527 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:57:42.524+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:42.526504+0000) 2019-10-18T05:57:43.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:43.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:43.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:43.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:43.412 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:43.405+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:44.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:44.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:44.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:44.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:45.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:45.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:45.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:45.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:46.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:46.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:46.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:46.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:46.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:57:46.297+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:46.302668+0000) 2019-10-18T05:57:47.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:47.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:47.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:47.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:48.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:48.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:48.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:48.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:48.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:57:48.333+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:48.339309+0000) 2019-10-18T05:57:48.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:48.405+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:49.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:49.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:49.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:49.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:50.034 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:57:50.016+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:50.019552+0000) 2019-10-18T05:57:50.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:50.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:50.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:50.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:51.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:51.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:51.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:51.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:52.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:52.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:52.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:52.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:52.528 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:57:52.524+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:52.526659+0000) 2019-10-18T05:57:53.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:53.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:53.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:53.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:53.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:53.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:54.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:54.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:54.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:54.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:55.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:55.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:55.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:55.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:56.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:57:56.301+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:56.302766+0000) 2019-10-18T05:57:56.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:56.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:56.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:56.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:57.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:57.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:57.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:57.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:58.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:57:58.337+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:57:58.339428+0000) 2019-10-18T05:57:58.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:58.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:58.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:58.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:57:58.415 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:57:58.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:57:59.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:59.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:57:59.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:57:59.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:00.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:58:00.016+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:00.019684+0000) 2019-10-18T05:58:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:00.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:00.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:00.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:01.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:01.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:01.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:01.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:02.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:02.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:02.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:02.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:02.528 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:02.524+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:02.526827+0000) 2019-10-18T05:58:02.529 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:02.524+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:02.527871+0000) 2019-10-18T05:58:03.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:03.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:03.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:03.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:03.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:03.413+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:04.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:04.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:04.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:04.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:05.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:05.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:05.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:05.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:06.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:58:06.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:06.302861+0000) 2019-10-18T05:58:06.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:06.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:06.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:06.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:07.028 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:07.024+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:07.027231+0000) 2019-10-18T05:58:07.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:07.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:07.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:07.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:08.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:08.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:08.339555+0000) 2019-10-18T05:58:08.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:08.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:08.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:08.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:08.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:08.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:09.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:09.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:09.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:09.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:10.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:58:10.016+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:10.019796+0000) 2019-10-18T05:58:10.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:10.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:10.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:10.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:11.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:11.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:11.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:11.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:12.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:12.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:12.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:12.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:13.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:13.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:13.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:13.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:13.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:13.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:14.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:14.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:14.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:14.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:15.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:15.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:15.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:15.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:16.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:16.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:16.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:16.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:16.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:58:16.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:16.302958+0000) 2019-10-18T05:58:17.028 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:17.024+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:17.027374+0000) 2019-10-18T05:58:17.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:17.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:17.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:17.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:18.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:18.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:18.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:18.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:18.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:18.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:18.339678+0000) 2019-10-18T05:58:18.418 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:18.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:19.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:19.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:19.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:19.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:20.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:58:20.016+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:20.019908+0000) 2019-10-18T05:58:20.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:20.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:20.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:20.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:21.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:21.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:21.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:21.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:22.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:22.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:22.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:22.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:23.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:23.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:23.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:23.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:23.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:24.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:24.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:24.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:24.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:25.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:25.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:25.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:25.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:26.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:26.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:26.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:26.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:58:26.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:26.303059+0000) 2019-10-18T05:58:27.028 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:27.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:27.027558+0000) 2019-10-18T05:58:27.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:27.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:27.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:27.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:28.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:28.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:28.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:28.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:28.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:28.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:28.339796+0000) 2019-10-18T05:58:28.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:28.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:29.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:29.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:29.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:29.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:30.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:58:30.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:30.020024+0000) 2019-10-18T05:58:30.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:30.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:30.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:30.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:31.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:31.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:31.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:31.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:32.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:32.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:32.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:32.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:33.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:33.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:33.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:33.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:33.421 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:33.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:34.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:34.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:34.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:34.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:35.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:35.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:35.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:35.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:36.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:36.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:36.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:36.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:36.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:58:36.300+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:36.303165+0000) 2019-10-18T05:58:37.029 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:37.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:37.027769+0000) 2019-10-18T05:58:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:37.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:37.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:37.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:38.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:38.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:38.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:38.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:38.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:38.336+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:38.339911+0000) 2019-10-18T05:58:38.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:38.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:39.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:39.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:40.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:58:40.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:40.020137+0000) 2019-10-18T05:58:40.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:40.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:40.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:40.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:41.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:41.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:41.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:41.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:42.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:42.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:42.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:42.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:43.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:43.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:43.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:43.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:43.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:43.415+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:44.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:44.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:44.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:44.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:45.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:45.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:45.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:45.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:46.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:46.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:46.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:46.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:46.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:58:46.299+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:46.303262+0000) 2019-10-18T05:58:47.029 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:47.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:47.027985+0000) 2019-10-18T05:58:47.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:47.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:47.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:47.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:48.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:48.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:48.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:48.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:48.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:48.335+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:48.340027+0000) 2019-10-18T05:58:48.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:48.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:49.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:49.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:49.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:49.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:50.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:58:50.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:50.020261+0000) 2019-10-18T05:58:50.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:50.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:50.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:50.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:51.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:51.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:51.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:51.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:52.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:52.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:52.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:52.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:53.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:53.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:53.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:53.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:53.424 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:53.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:54.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:54.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:54.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:54.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:55.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:55.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:55.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:55.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:56.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:56.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:56.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:56.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:56.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:58:56.299+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:56.303364+0000) 2019-10-18T05:58:57.029 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:58:57.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:57.028166+0000) 2019-10-18T05:58:57.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:57.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:57.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:57.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:58.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:58.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:58.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:58.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:58:58.341 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:58.335+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:58.340144+0000) 2019-10-18T05:58:58.342 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:58:58.335+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:58:58.341175+0000) 2019-10-18T05:58:58.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:58:58.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:58:59.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:59.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:58:59.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:58:59.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:00.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:59:00.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:00.020362+0000) 2019-10-18T05:59:00.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:00.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:00.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:00.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:01.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:01.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:01.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:01.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:02.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:02.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:02.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:02.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:02.841 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:59:02.835+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:02.840330+0000) 2019-10-18T05:59:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:03.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:03.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:03.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:03.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:04.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:04.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:04.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:04.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:05.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:05.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:05.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:05.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:06.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:06.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:06.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:06.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:06.304 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:06.299+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:06.303461+0000) 2019-10-18T05:59:07.029 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:59:07.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:07.028351+0000) 2019-10-18T05:59:07.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:07.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:07.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:07.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:08.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:08.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:08.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:08.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:08.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:08.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:09.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:09.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:09.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:09.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:10.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:59:10.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:10.020476+0000) 2019-10-18T05:59:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:10.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:10.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:11.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:11.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:11.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:11.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:12.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:12.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:12.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:12.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:12.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:59:12.835+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:12.840440+0000) 2019-10-18T05:59:13.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:13.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:13.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:13.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:13.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:13.423+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:14.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:14.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:14.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:15.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:15.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:15.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:15.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:16.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:16.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:16.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:16.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:16.305 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:16.298+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:16.303553+0000) 2019-10-18T05:59:17.029 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:59:17.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:17.028532+0000) 2019-10-18T05:59:17.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:17.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:17.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:17.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:18.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:18.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:18.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:18.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:18.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:18.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:19.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:19.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:19.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:19.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:20.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:59:20.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:20.020591+0000) 2019-10-18T05:59:20.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:20.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:20.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:20.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:21.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:21.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:21.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:21.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:22.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:22.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:22.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:22.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:22.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:59:22.834+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:22.840549+0000) 2019-10-18T05:59:23.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:23.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:23.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:23.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:23.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:23.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:24.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:24.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:24.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:24.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:25.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:25.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:25.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:25.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:26.305 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:26.298+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:26.303650+0000) 2019-10-18T05:59:26.306 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:26.298+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:26.304780+0000) 2019-10-18T05:59:26.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:26.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:26.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:26.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:27.030 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:59:27.023+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:27.028720+0000) 2019-10-18T05:59:27.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:27.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:27.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:27.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:28.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:28.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:28.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:28.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:28.430 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:28.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:29.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:29.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:29.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:29.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:30.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:59:30.015+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:30.020707+0000) 2019-10-18T05:59:30.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:30.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:30.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:30.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:30.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:30.798+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:30.803851+0000) 2019-10-18T05:59:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:31.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:31.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:32.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:32.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:32.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:32.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:32.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:59:32.834+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:32.840656+0000) 2019-10-18T05:59:33.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:33.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:33.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:33.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:33.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:33.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:34.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:34.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:34.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:34.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:35.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:35.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:35.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:35.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:36.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:36.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:36.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:36.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:37.030 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:59:37.022+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:37.028905+0000) 2019-10-18T05:59:37.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:37.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:37.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:37.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:38.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:38.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:38.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:38.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:38.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:38.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:39.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:39.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:39.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:39.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:40.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:59:40.014+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:40.020832+0000) 2019-10-18T05:59:40.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:40.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:40.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:40.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:40.805 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:40.798+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:40.803954+0000) 2019-10-18T05:59:41.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:41.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:41.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:41.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:42.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:42.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:42.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:42.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:42.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:59:42.834+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:42.840764+0000) 2019-10-18T05:59:43.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:43.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:43.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:43.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:43.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:43.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:44.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:44.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:44.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:44.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:45.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:45.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:45.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:45.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:46.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:46.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:46.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:46.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:47.030 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:59:47.022+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:47.029093+0000) 2019-10-18T05:59:47.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:47.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:47.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:47.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:48.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:48.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:48.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:48.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:48.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:48.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:49.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:49.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:49.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:49.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:50.022 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T05:59:50.014+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:50.020947+0000) 2019-10-18T05:59:50.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:50.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:50.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:50.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:50.805 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T05:59:50.797+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:50.804059+0000) 2019-10-18T05:59:51.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:51.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:51.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:51.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:52.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:52.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:52.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:52.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:52.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T05:59:52.833+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:52.840877+0000) 2019-10-18T05:59:53.435 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:53.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:53.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:53.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:53.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:53.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:54.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:54.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:54.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:54.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:55.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:55.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:55.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:55.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:56.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:56.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:56.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:56.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:57.030 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T05:59:57.022+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T04:59:57.029273+0000) 2019-10-18T05:59:57.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:57.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:57.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:57.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:58.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T05:59:58.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T05:59:58.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:58.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:58.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:58.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T05:59:59.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:59.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T05:59:59.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T05:59:59.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:00.022 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:00:00.014+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:00.021061+0000) 2019-10-18T06:00:00.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:00.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:00.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:00.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:00.805 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:00:00.797+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:00.804161+0000) 2019-10-18T06:00:01.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:01.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:01.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:01.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:02.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:02.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:02.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:02.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:02.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:00:02.833+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:02.840983+0000) 2019-10-18T06:00:03.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:03.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:03.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:03.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:03.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:03.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:04.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:04.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:04.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:04.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:05.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:05.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:05.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:05.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:06.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:06.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:06.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:06.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:07.030 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:00:07.022+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:07.029483+0000) 2019-10-18T06:00:07.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:07.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:07.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:07.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:08.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:08.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:08.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:08.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:08.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:08.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:09.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:09.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:09.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:09.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:10.022 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:00:10.014+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:10.021177+0000) 2019-10-18T06:00:10.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:10.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:10.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:10.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:10.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:00:10.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:10.804266+0000) 2019-10-18T06:00:11.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:11.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:11.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:11.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:12.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:12.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:12.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:12.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:12.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:00:12.837+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:12.841090+0000) 2019-10-18T06:00:13.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:13.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:13.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:13.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:13.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:13.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:14.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:14.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:14.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:14.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:15.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:15.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:15.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:16.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:16.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:16.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:16.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:17.030 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:00:17.022+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:17.029681+0000) 2019-10-18T06:00:17.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:17.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:17.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:17.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:18.439 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:18.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:18.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:18.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:18.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:18.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:19.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:19.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:19.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:19.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:20.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:00:20.014+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:20.021296+0000) 2019-10-18T06:00:20.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:20.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:20.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:20.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:20.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:00:20.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:20.804374+0000) 2019-10-18T06:00:21.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:21.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:21.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:21.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:22.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:22.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:22.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:22.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:22.842 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:00:22.837+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:22.841203+0000) 2019-10-18T06:00:23.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:23.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:23.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:23.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:23.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:23.437+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:24.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:24.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:24.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:24.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:25.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:25.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:25.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:25.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:26.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:26.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:26.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:26.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:27.031 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:00:27.022+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:27.029859+0000) 2019-10-18T06:00:27.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:27.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:27.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:27.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:28.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:28.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:28.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:28.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:28.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:28.436+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:29.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:29.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:29.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:29.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:30.022 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:00:30.018+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:30.021412+0000) 2019-10-18T06:00:30.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:30.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:30.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:30.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:30.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:00:30.800+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:30.804479+0000) 2019-10-18T06:00:31.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:31.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:31.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:31.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:32.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:32.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:32.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:32.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:32.843 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:00:32.836+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:32.841306+0000) 2019-10-18T06:00:33.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:33.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:33.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:33.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:33.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:33.436+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:34.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:34.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:34.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:34.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:35.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:35.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:35.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:35.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:36.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:36.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:36.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:36.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:37.031 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:00:37.026+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:37.030041+0000) 2019-10-18T06:00:37.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:37.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:37.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:37.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:38.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:38.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:38.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:38.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:38.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:38.436+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:39.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:39.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:39.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:39.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:40.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:00:40.018+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:40.021559+0000) 2019-10-18T06:00:40.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:40.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:40.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:40.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:40.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:00:40.800+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:40.804584+0000) 2019-10-18T06:00:41.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:41.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:41.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:41.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:42.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:42.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:42.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:42.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:42.843 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:00:42.836+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:42.841411+0000) 2019-10-18T06:00:43.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:43.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T05:50:08.923009+0000 (oldest deadline 2019-10-18T05:50:28.923009+0000) 2019-10-18T06:00:43.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:43.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:43.443 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:43.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:44.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:44.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:45.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:45.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:46.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:46.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:47.031 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:00:47.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:47.030231+0000) 2019-10-18T06:00:47.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:47.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:48.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:48.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:48.444 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:48.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:49.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:49.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:50.022 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:00:50.017+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:50.021670+0000) 2019-10-18T06:00:50.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:50.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:50.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:00:50.800+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:50.804691+0000) 2019-10-18T06:00:51.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:51.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:52.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:52.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:52.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:00:52.840+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:52.841522+0000) 2019-10-18T06:00:53.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:53.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:53.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:53.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:54.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:54.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:55.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:55.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:56.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:56.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:57.032 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:00:57.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:00:57.030439+0000) 2019-10-18T06:00:57.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:57.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:58.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:58.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:00:58.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:00:58.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:00:59.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:00:59.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:00.025 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:01:00.017+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:00.021810+0000) 2019-10-18T06:01:00.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:00.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:00.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:01:00.799+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:00.804801+0000) 2019-10-18T06:01:01.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:01.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:02.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:02.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:02.843 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:01:02.839+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:02.841629+0000) 2019-10-18T06:01:03.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:03.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:03.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:03.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:04.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:04.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:04.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:04.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:05.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:05.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:05.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:05.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:06.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:06.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:06.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:06.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:07.031 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:01:07.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:07.030607+0000) 2019-10-18T06:01:07.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:07.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:07.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:07.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:08.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:08.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:08.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:08.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:08.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:08.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:09.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:09.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:09.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:09.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:10.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:01:10.017+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:10.021948+0000) 2019-10-18T06:01:10.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:10.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:10.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:10.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:10.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:01:10.799+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:10.804918+0000) 2019-10-18T06:01:11.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:11.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:11.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:11.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:12.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:12.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:12.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:12.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:12.843 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:01:12.839+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:12.841737+0000) 2019-10-18T06:01:13.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:13.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:13.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:13.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:13.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:13.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:14.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:14.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:14.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:14.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:15.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:15.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:15.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:15.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:16.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:16.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:16.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:16.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:17.032 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:01:17.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:17.030790+0000) 2019-10-18T06:01:17.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:17.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:17.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:17.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:18.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:18.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:18.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:18.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:18.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:18.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:19.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:19.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:19.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:19.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:20.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:01:20.017+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:20.022095+0000) 2019-10-18T06:01:20.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:20.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:20.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:20.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:20.806 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:01:20.799+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:20.805025+0000) 2019-10-18T06:01:21.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:21.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:21.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:21.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:22.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:22.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:22.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:22.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:22.843 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:01:22.839+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:22.841842+0000) 2019-10-18T06:01:23.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:23.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:23.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:23.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:23.450 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:23.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:24.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:24.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:24.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:24.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:25.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:25.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:25.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:25.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:26.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:26.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:26.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:26.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:27.032 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:01:27.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:27.031002+0000) 2019-10-18T06:01:27.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:27.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:27.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:27.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:28.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:28.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:28.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:28.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:28.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:28.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:29.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:29.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:29.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:29.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:30.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:30.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:30.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:30.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:30.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:01:30.021+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:30.022230+0000) 2019-10-18T06:01:30.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:01:30.799+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:30.805134+0000) 2019-10-18T06:01:30.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:30.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:30.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:30.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:32.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:31.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:32.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:31.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:32.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:01:32.839+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:32.841948+0000) 2019-10-18T06:01:33.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:33.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:33.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:33.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:33.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:33.447+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:33.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:33.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:33.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:33.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:34.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:34.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:34.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:34.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:35.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:35.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:35.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:35.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:36.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:36.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:36.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:36.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:37.040 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:01:37.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:37.031219+0000) 2019-10-18T06:01:37.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:37.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:37.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:37.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:38.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:38.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:38.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:38.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:38.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:38.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:39.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:39.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:39.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:39.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:40.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:01:40.021+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:40.022361+0000) 2019-10-18T06:01:40.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:01:40.798+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:40.805239+0000) 2019-10-18T06:01:40.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:40.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:40.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:40.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:42.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:42.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:42.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:42.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:42.843 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:01:42.838+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:42.842056+0000) 2019-10-18T06:01:43.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:43.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:43.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:43.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:43.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:43.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:44.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:44.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:44.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:44.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:45.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:45.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:45.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:45.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:46.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:46.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:46.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:46.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:47.032 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:01:47.029+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:47.031427+0000) 2019-10-18T06:01:47.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:47.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:47.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:47.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:48.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:48.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:48.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:48.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:48.454 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:48.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:49.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:49.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:49.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:49.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:50.023 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:01:50.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:50.022495+0000) 2019-10-18T06:01:50.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:50.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:50.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:50.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:50.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:01:50.798+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:50.805343+0000) 2019-10-18T06:01:51.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:51.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:51.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:51.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:52.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:52.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:52.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:52.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:52.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:01:52.838+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:52.842164+0000) 2019-10-18T06:01:53.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:53.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:53.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:53.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:53.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:53.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:54.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:54.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:54.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:54.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:55.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:55.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:55.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:55.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:56.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:56.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:56.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:56.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:57.032 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:01:57.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:01:57.031608+0000) 2019-10-18T06:01:57.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:57.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:57.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:57.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:58.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:58.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:58.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:58.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:01:58.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:01:58.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:01:59.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:59.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:01:59.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:01:59.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:00.026 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:02:00.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:00.022646+0000) 2019-10-18T06:02:00.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:00.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:00.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:00.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:00.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:02:00.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:00.805454+0000) 2019-10-18T06:02:01.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:01.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:01.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:01.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:02.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:02.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:02.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:02.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:02.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:02.838+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:02.842268+0000) 2019-10-18T06:02:03.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:03.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:03.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:03.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:03.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:03.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:04.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:04.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:04.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:04.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:05.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:05.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:05.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:05.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:06.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:06.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:06.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:06.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:07.033 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:02:07.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:07.031820+0000) 2019-10-18T06:02:07.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:07.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:07.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:07.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:08.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:08.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:08.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:08.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:08.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:08.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:09.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:09.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:09.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:09.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:10.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:02:10.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:10.022804+0000) 2019-10-18T06:02:10.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:10.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:10.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:10.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:10.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:02:10.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:10.805555+0000) 2019-10-18T06:02:11.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:11.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:11.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:11.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:12.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:12.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:12.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:12.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:12.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:12.837+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:12.842376+0000) 2019-10-18T06:02:13.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:13.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:13.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:13.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:13.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:13.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:14.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:14.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:14.165+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:15.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:15.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:15.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:15.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:16.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:16.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:16.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:16.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:17.033 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:02:17.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:17.032002+0000) 2019-10-18T06:02:17.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:17.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:17.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:17.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:18.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:18.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:18.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:18.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:18.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:18.453+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:19.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:19.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:19.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:19.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:20.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:02:20.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:20.022965+0000) 2019-10-18T06:02:20.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:20.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:20.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:20.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:20.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:02:20.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:20.805652+0000) 2019-10-18T06:02:21.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:21.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:21.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:21.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:22.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:22.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:22.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:22.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:22.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:22.837+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:22.842485+0000) 2019-10-18T06:02:23.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:23.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:23.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:23.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:23.460 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:23.453+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:24.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:24.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:24.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:24.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:25.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:25.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:25.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:25.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:26.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:26.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:26.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:26.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:27.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:02:27.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:27.032185+0000) 2019-10-18T06:02:27.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:27.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:27.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:27.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:28.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:28.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:28.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:28.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:28.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:28.457+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:29.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:29.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:29.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:29.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:30.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:02:30.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:30.023112+0000) 2019-10-18T06:02:30.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:30.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:30.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:30.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:30.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:02:30.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:30.805753+0000) 2019-10-18T06:02:31.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:31.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:31.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:31.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:32.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:32.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:32.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:32.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:32.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:32.841+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:32.842592+0000) 2019-10-18T06:02:33.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:33.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:33.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:33.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:33.462 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:33.457+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:34.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:34.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:34.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:34.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:35.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:35.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:35.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:35.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:36.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:36.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:36.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:36.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:37.033 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:02:37.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:37.032367+0000) 2019-10-18T06:02:37.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:37.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:37.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:37.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:38.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:38.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:38.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:38.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:38.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:38.457+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:39.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:39.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:39.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:39.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:40.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:02:40.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:40.023251+0000) 2019-10-18T06:02:40.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:40.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:40.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:40.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:40.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:02:40.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:40.805855+0000) 2019-10-18T06:02:41.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:41.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:41.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:41.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:42.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:42.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:42.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:42.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:42.844 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:42.841+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:42.842695+0000) 2019-10-18T06:02:43.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:43.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:43.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:43.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:43.463 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:43.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:44.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:44.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:44.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:44.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:45.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:45.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:45.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:45.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:46.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:46.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:46.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:46.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:47.033 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:02:47.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:47.032549+0000) 2019-10-18T06:02:47.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:47.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:47.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:47.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:48.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:48.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:48.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:48.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:48.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:48.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:49.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:49.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:49.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:49.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:50.024 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:02:50.020+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:50.023359+0000) 2019-10-18T06:02:50.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:50.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:50.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:50.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:50.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:02:50.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:50.805956+0000) 2019-10-18T06:02:51.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:51.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:51.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:51.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:52.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:52.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:52.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:52.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:52.845 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:52.840+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:52.842796+0000) 2019-10-18T06:02:52.845 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:52.840+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:52.844085+0000) 2019-10-18T06:02:53.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:53.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:53.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:53.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:53.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:53.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:54.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:54.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:54.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:54.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:55.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:55.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:55.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:55.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:56.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:56.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:56.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:57.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:02:57.028+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:57.032736+0000) 2019-10-18T06:02:57.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:57.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:57.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:57.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:57.344 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:02:57.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:02:57.343000+0000) 2019-10-18T06:02:58.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:58.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:58.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:58.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:02:58.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:02:58.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:02:59.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:59.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:02:59.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:02:59.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:00.026 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:00.019+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:00.023513+0000) 2019-10-18T06:03:00.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:00.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:00.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:00.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:00.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:03:00.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:00.806062+0000) 2019-10-18T06:03:01.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:01.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:01.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:01.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:02.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:02.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:02.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:02.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:03.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:03.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:03.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:03.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:03.467 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:03.464+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:04.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:04.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:04.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:04.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:05.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:05.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:05.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:05.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:06.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:06.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:06.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:06.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:07.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:03:07.027+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:07.032883+0000) 2019-10-18T06:03:07.344 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:03:07.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:07.343099+0000) 2019-10-18T06:03:07.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:07.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:07.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:07.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:08.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:08.464+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:08.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:08.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:08.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:08.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:09.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:09.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:09.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:09.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:10.025 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:10.019+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:10.023681+0000) 2019-10-18T06:03:10.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:10.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:10.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:10.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:10.807 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:03:10.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:10.806163+0000) 2019-10-18T06:03:11.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:11.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:11.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:11.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:12.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:12.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:12.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:12.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:13.469 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:13.464+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:13.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:13.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:13.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:13.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:14.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:14.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:14.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:14.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:15.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:15.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:15.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:15.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:16.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:16.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:16.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:16.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:17.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:03:17.027+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:17.033024+0000) 2019-10-18T06:03:17.347 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:03:17.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:17.343206+0000) 2019-10-18T06:03:17.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:17.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:17.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:17.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:18.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:18.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:18.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:18.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:18.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:18.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:19.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:19.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:19.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:19.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:20.025 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:20.019+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:20.023840+0000) 2019-10-18T06:03:20.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:20.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:20.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:20.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:20.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:03:20.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:20.806265+0000) 2019-10-18T06:03:21.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:21.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:21.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:21.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:22.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:22.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:22.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:23.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:23.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:23.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:23.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:23.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:23.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:24.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:24.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:25.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:25.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:25.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:25.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:26.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:26.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:26.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:26.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:27.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:03:27.027+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:27.033210+0000) 2019-10-18T06:03:27.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:03:27.339+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:27.343305+0000) 2019-10-18T06:03:27.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:27.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:27.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:27.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:28.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:28.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:28.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:28.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:28.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:28.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:29.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:29.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:29.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:29.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:30.025 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:30.019+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:30.023985+0000) 2019-10-18T06:03:30.026 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:30.019+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:30.024984+0000) 2019-10-18T06:03:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:30.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:30.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:30.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:30.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:03:30.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:30.806368+0000) 2019-10-18T06:03:31.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:31.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:31.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:31.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:32.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:32.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:32.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:32.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:33.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:33.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:33.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:33.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:33.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:33.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:34.525 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:34.519+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:34.524310+0000) 2019-10-18T06:03:34.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:34.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:34.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:34.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:35.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:35.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:35.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:35.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:36.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:36.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:36.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:36.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:37.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:03:37.027+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:37.033390+0000) 2019-10-18T06:03:37.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:03:37.339+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:37.343409+0000) 2019-10-18T06:03:37.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:37.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:37.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:37.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:38.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:38.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:38.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:38.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:38.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:38.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:39.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:39.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:39.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:39.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:40.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:40.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:40.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:40.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:40.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:03:40.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:40.806473+0000) 2019-10-18T06:03:41.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:41.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:41.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:41.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:42.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:42.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:42.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:42.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:43.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:43.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:43.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:43.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:43.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:43.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:44.525 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:44.519+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:44.524493+0000) 2019-10-18T06:03:44.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:44.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:44.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:44.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:45.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:45.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:45.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:45.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:46.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:46.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:46.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:46.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:47.034 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:03:47.027+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:47.033570+0000) 2019-10-18T06:03:47.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:03:47.339+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:47.343508+0000) 2019-10-18T06:03:47.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:47.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:47.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:47.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:48.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:48.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:48.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:48.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:48.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:48.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:49.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:49.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:49.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:49.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:50.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:50.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:50.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:50.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:50.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:03:50.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:50.806577+0000) 2019-10-18T06:03:51.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:51.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:51.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:51.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:52.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:52.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:52.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:52.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:53.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:53.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:53.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:53.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:53.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:53.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:54.526 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:03:54.519+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:54.524686+0000) 2019-10-18T06:03:54.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:54.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:54.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:54.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:55.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:55.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:55.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:55.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:56.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:56.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:56.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:56.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:57.035 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:03:57.027+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:57.033758+0000) 2019-10-18T06:03:57.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:03:57.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:03:57.343612+0000) 2019-10-18T06:03:57.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:57.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:57.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:57.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:58.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:03:58.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:03:58.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:58.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:58.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:58.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:03:59.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:59.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:03:59.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:03:59.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:00.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:00.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:00.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:00.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:00.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:04:00.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:00.806686+0000) 2019-10-18T06:04:01.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:01.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:01.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:01.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:02.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:02.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:02.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:02.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:03.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:03.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:03.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:03.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:03.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:03.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:04.526 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:04:04.519+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:04.524900+0000) 2019-10-18T06:04:04.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:04.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:04.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:04.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:05.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:05.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:05.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:05.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:06.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:06.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:06.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:06.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:07.035 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:07.026+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:07.033941+0000) 2019-10-18T06:04:07.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:04:07.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:07.343717+0000) 2019-10-18T06:04:07.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:07.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:07.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:07.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:08.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:08.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:08.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:08.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:08.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:08.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:09.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:09.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:09.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:09.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:10.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:04:10.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:10.806790+0000) 2019-10-18T06:04:10.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:10.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:10.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:10.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:11.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:11.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:11.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:11.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:12.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:12.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:12.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:12.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:13.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:13.474+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:13.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:13.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:13.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:13.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:14.526 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:04:14.518+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:14.525111+0000) 2019-10-18T06:04:14.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:14.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:14.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:14.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:15.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:15.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:15.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:15.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:16.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:16.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:16.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:16.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:17.035 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:17.026+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:17.034084+0000) 2019-10-18T06:04:17.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:04:17.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:17.343822+0000) 2019-10-18T06:04:17.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:17.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:17.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:17.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:18.480 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:18.474+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:18.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:18.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:18.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:18.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:19.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:19.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:19.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:19.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:20.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:04:20.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:20.806892+0000) 2019-10-18T06:04:20.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:20.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:20.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:20.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:21.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:21.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:21.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:21.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:22.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:22.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:22.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:22.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:23.481 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:23.474+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:23.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:23.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:23.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:23.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:24.526 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:04:24.518+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:24.525331+0000) 2019-10-18T06:04:24.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:24.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:24.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:24.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:26.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:25.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:26.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:25.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:27.035 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:27.030+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:27.034321+0000) 2019-10-18T06:04:27.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:27.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:27.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:27.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:27.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:04:27.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:27.343927+0000) 2019-10-18T06:04:28.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:28.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:28.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:28.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:28.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:28.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:29.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:29.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:29.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:29.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:30.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:30.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:30.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:30.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:30.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:04:30.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:30.807006+0000) 2019-10-18T06:04:31.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:31.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:31.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:31.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:31.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:31.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:31.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:31.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:33.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:33.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:33.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:33.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:33.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:33.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:34.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:33.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:34.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:33.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:34.526 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:04:34.518+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:34.525545+0000) 2019-10-18T06:04:35.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:35.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:35.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:35.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:36.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:36.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:36.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:36.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:37.035 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:37.030+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:37.034501+0000) 2019-10-18T06:04:37.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:37.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:37.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:37.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:37.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:04:37.337+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:37.344034+0000) 2019-10-18T06:04:38.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:38.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:38.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:38.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:38.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:38.477+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:39.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:39.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:39.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:39.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:40.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:39.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:40.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:39.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:40.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:04:40.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:40.807122+0000) 2019-10-18T06:04:40.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:40.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:40.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:40.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:42.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:41.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:42.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:41.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:42.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:42.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:42.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:42.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:43.484 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:43.477+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:43.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:43.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:43.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:43.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:44.527 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:04:44.518+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:44.525731+0000) 2019-10-18T06:04:44.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:44.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:44.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:44.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:45.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:45.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:45.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:45.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:46.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:46.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:46.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:46.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:47.036 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:47.030+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:47.034676+0000) 2019-10-18T06:04:47.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:04:47.337+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:47.344138+0000) 2019-10-18T06:04:47.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:47.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:47.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:47.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:48.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:48.481+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:48.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:48.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:48.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:48.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:49.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:49.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:49.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:49.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:50.808 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:04:50.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:50.807226+0000) 2019-10-18T06:04:50.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:50.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:50.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:50.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:51.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:51.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:51.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:51.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:52.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:52.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:52.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:52.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:53.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:53.481+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:53.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:53.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:53.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:53.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:54.527 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:04:54.518+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:54.525914+0000) 2019-10-18T06:04:55.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:55.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:55.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:55.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:56.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:56.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:56.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:56.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:57.036 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:57.030+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:57.034876+0000) 2019-10-18T06:04:57.036 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:04:57.030+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:57.035777+0000) 2019-10-18T06:04:57.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:57.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:57.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:57.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:57.345 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:04:57.341+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:04:57.344242+0000) 2019-10-18T06:04:58.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:58.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:58.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:58.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:04:58.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:04:58.485+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:04:59.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:59.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:04:59.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:04:59.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:00.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:00.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:00.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:00.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:00.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:05:00.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:00.807334+0000) 2019-10-18T06:05:01.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:01.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:01.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:01.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:01.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:05:01.530+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:01.535249+0000) 2019-10-18T06:05:02.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:02.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:02.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:02.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:02.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:02.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:02.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:03.488 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:03.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:03.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:03.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:03.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:03.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:04.527 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:04.522+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:04.526077+0000) 2019-10-18T06:05:04.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:04.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:04.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:04.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:05.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:05.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:05.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:05.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:06.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:06.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:06.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:06.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:07.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:05:07.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:07.344346+0000) 2019-10-18T06:05:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:07.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:07.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:08.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:08.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:08.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:08.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:08.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:08.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:09.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:09.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:09.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:09.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:10.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:05:10.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:10.807437+0000) 2019-10-18T06:05:10.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:10.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:10.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:10.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:11.539 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:05:11.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:11.535402+0000) 2019-10-18T06:05:11.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:11.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:11.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:11.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:12.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:12.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:12.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:12.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:13.491 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:13.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:13.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:13.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:13.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:13.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:14.528 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:14.525+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:14.526256+0000) 2019-10-18T06:05:14.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:14.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:14.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:14.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:15.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:15.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:15.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:15.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:16.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:16.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:16.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:16.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:17.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:05:17.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:17.344452+0000) 2019-10-18T06:05:17.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:17.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:17.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:17.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:18.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:18.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:18.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:18.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:18.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:18.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:19.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:19.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:19.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:19.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:20.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:20.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:20.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:20.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:20.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:05:20.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:20.807538+0000) 2019-10-18T06:05:21.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:05:21.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:21.535585+0000) 2019-10-18T06:05:21.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:21.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:21.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:21.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:22.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:22.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:22.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:22.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:23.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:23.488+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:23.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:23.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:23.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:23.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:24.528 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:24.525+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:24.526428+0000) 2019-10-18T06:05:24.528 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:24.525+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:24.527402+0000) 2019-10-18T06:05:24.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:24.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:24.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:24.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:25.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:25.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:25.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:25.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:26.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:26.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:26.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:26.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:27.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:05:27.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:27.344551+0000) 2019-10-18T06:05:27.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:27.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:27.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:27.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:28.492 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:28.488+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:28.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:28.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:28.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:28.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:29.028 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:29.025+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:29.026905+0000) 2019-10-18T06:05:29.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:29.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:29.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:29.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:30.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:30.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:30.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:30.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:30.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:05:30.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:30.807645+0000) 2019-10-18T06:05:31.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:05:31.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:31.535781+0000) 2019-10-18T06:05:31.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:31.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:31.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:31.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:32.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:32.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:32.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:32.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:33.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:33.488+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:33.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:33.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:33.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:33.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:34.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:34.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:34.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:34.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:35.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:35.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:35.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:35.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:36.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:36.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:37.357 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:05:37.340+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:37.344656+0000) 2019-10-18T06:05:37.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:37.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:37.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:37.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:38.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:38.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:38.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:38.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:38.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:38.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:39.028 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:39.025+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:39.027089+0000) 2019-10-18T06:05:39.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:39.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:39.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:39.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:40.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:40.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:40.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:40.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:40.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:05:40.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:40.807747+0000) 2019-10-18T06:05:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:41.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:41.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:41.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:41.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:05:41.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:41.535976+0000) 2019-10-18T06:05:42.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:42.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:42.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:42.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:43.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:43.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:43.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:43.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:43.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:43.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:44.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:44.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:44.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:44.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:45.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:45.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:45.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:45.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:46.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:46.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:46.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:46.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:47.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:05:47.339+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:47.344769+0000) 2019-10-18T06:05:47.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:47.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:47.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:47.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:48.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:48.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:48.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:48.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:48.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:48.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:49.028 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:49.025+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:49.027270+0000) 2019-10-18T06:05:49.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:49.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:49.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:49.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:50.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:50.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:50.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:50.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:50.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:05:50.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:50.807854+0000) 2019-10-18T06:05:51.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:05:51.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:51.536113+0000) 2019-10-18T06:05:51.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:51.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:51.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:51.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:52.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:52.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:52.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:52.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:53.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:53.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:53.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:53.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:53.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:53.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:54.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:54.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:54.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:55.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:55.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:55.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:55.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:56.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:56.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:57.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:05:57.339+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:57.344869+0000) 2019-10-18T06:05:57.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:57.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:57.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:57.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:58.498 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:05:58.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:05:58.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:58.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:58.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:58.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:05:59.028 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:05:59.025+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:05:59.027454+0000) 2019-10-18T06:05:59.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:59.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:05:59.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:05:59.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:00.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:00.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:00.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:00.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:00.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:06:00.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:00.807958+0000) 2019-10-18T06:06:01.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:06:01.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:01.536300+0000) 2019-10-18T06:06:01.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:01.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:01.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:01.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:02.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:02.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:02.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:02.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:03.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:03.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:03.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:03.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:03.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:03.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:04.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:04.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:04.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:04.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:05.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:05.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:05.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:05.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:06.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:06.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:06.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:06.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:07.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:06:07.339+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:07.344966+0000) 2019-10-18T06:06:07.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:07.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:07.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:07.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:08.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:08.495+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:08.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:08.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:08.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:08.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:09.029 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:06:09.025+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:09.027654+0000) 2019-10-18T06:06:09.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:09.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:09.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:09.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:10.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:10.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:10.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:10.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:10.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:06:10.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:10.808061+0000) 2019-10-18T06:06:11.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:11.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:11.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:11.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:11.537 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:06:11.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:11.536501+0000) 2019-10-18T06:06:12.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:12.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:12.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:12.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:13.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:13.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:13.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:13.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:13.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:13.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:14.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:14.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:14.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:14.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:15.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:15.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:15.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:15.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:16.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:16.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:16.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:16.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:17.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:06:17.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:17.345069+0000) 2019-10-18T06:06:17.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:17.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:17.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:17.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:18.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:18.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:18.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:18.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:18.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:18.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:19.029 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:06:19.025+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:19.027839+0000) 2019-10-18T06:06:19.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:19.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:19.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:19.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:20.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:20.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:20.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:20.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:20.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:06:20.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:20.808162+0000) 2019-10-18T06:06:21.538 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:06:21.533+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:21.536687+0000) 2019-10-18T06:06:21.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:21.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:21.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:21.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:22.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:22.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:22.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:22.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:23.502 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:23.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:23.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:23.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:23.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:23.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:24.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:24.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:24.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:24.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:25.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:25.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:25.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:25.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:26.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:26.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:26.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:26.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:27.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:06:27.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:27.345174+0000) 2019-10-18T06:06:27.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:27.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:27.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:27.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:28.502 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:28.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:28.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:28.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:28.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:28.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:29.029 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:06:29.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:29.028019+0000) 2019-10-18T06:06:29.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:29.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:29.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:29.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:30.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:30.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:30.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:30.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:30.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:06:30.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:30.808262+0000) 2019-10-18T06:06:31.538 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:06:31.532+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:31.536876+0000) 2019-10-18T06:06:31.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:31.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:31.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:31.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:32.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:32.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:32.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:32.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:33.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:33.498+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:33.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:33.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:33.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:33.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:34.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:34.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:34.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:34.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:35.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:35.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:35.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:35.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:36.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:36.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:36.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:36.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:37.346 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:06:37.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:37.345274+0000) 2019-10-18T06:06:37.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:37.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:37.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:37.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:38.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:38.498+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:38.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:38.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:38.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:38.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:39.029 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:06:39.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:39.028201+0000) 2019-10-18T06:06:39.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:39.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:39.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:39.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:40.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:40.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:40.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:40.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:40.809 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:06:40.802+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:40.808362+0000) 2019-10-18T06:06:41.538 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:06:41.532+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:41.537060+0000) 2019-10-18T06:06:41.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:41.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:41.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:41.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:42.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:42.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:42.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:42.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:43.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:43.498+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:43.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:43.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:43.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:43.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:44.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:44.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:44.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:44.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:45.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:45.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:45.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:45.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:46.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:46.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:46.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:46.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:47.347 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:06:47.338+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:47.345378+0000) 2019-10-18T06:06:47.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:47.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:47.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:47.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:48.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:48.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:48.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:48.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:48.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:48.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:49.029 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:06:49.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:49.028384+0000) 2019-10-18T06:06:49.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:49.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:49.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:49.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:50.813 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:06:50.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:50.808462+0000) 2019-10-18T06:06:50.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:50.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:50.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:50.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:51.538 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:06:51.532+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:51.537241+0000) 2019-10-18T06:06:51.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:51.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:51.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:51.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:52.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:52.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:52.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:52.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:53.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:53.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:53.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:53.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:53.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:53.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:54.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:54.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:54.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:54.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:55.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:55.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:55.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:55.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:56.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:56.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:56.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:56.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:57.348 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:06:57.341+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:57.345463+0000) 2019-10-18T06:06:57.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:57.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:57.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:57.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:58.508 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:06:58.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:06:59.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:59.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:06:59.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:06:59.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:06:59.029 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:06:59.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:06:59.028566+0000) 2019-10-18T06:07:00.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:00.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:00.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:00.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:00.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:07:00.801+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:00.808561+0000) 2019-10-18T06:07:01.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:01.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:01.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:01.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:01.538 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:07:01.532+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:01.537426+0000) 2019-10-18T06:07:02.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:02.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:02.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:02.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:03.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:03.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:03.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:03.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:03.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:03.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:04.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:04.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:04.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:04.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:05.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:05.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:05.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:05.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:06.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:06.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:06.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:06.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:07.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:07.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:07.347 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:07.345+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:07.345567+0000) 2019-10-18T06:07:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:08.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:08.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:08.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:08.505+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:09.030 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:07:09.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:09.028747+0000) 2019-10-18T06:07:09.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:09.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:09.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:10.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:10.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:10.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:10.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:10.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:07:10.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:10.808662+0000) 2019-10-18T06:07:11.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:11.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:11.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:11.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:11.541 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:07:11.532+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:11.537603+0000) 2019-10-18T06:07:12.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:12.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:12.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:12.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:13.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:13.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:13.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:13.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:13.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:13.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:14.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:14.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:14.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:14.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:15.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:15.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:15.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:15.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:16.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:16.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:16.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:16.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:17.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:17.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:17.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:17.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:17.347 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:17.345+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:17.345670+0000) 2019-10-18T06:07:18.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:18.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:18.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:18.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:18.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:18.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:19.030 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:07:19.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:19.028930+0000) 2019-10-18T06:07:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:19.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:19.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:20.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:20.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:20.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:20.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:20.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:07:20.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:20.808760+0000) 2019-10-18T06:07:21.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:21.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:21.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:21.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:21.539 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:07:21.532+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:21.537787+0000) 2019-10-18T06:07:22.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:22.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:22.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:22.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:23.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:23.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:23.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:23.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:23.512 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:23.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:24.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:24.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:24.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:24.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:25.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:25.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:25.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:25.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:26.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:26.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:26.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:26.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:27.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:27.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:27.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:27.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:27.347 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:27.344+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:27.345764+0000) 2019-10-18T06:07:28.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:28.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:28.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:28.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:28.513 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:28.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:29.030 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:07:29.024+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:29.029120+0000) 2019-10-18T06:07:29.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:29.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:29.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:29.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:30.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:30.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:30.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:30.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:30.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:07:30.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:30.808862+0000) 2019-10-18T06:07:31.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:31.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:31.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:31.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:31.539 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:07:31.531+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:31.537980+0000) 2019-10-18T06:07:32.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:32.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:32.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:32.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:33.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:33.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:33.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:33.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:33.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:33.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:34.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:34.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:34.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:34.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:35.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:35.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:35.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:35.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:36.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:36.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:36.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:37.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:37.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:37.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:37.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:37.348 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:37.344+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:37.345863+0000) 2019-10-18T06:07:37.349 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:37.344+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:37.346814+0000) 2019-10-18T06:07:38.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:38.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:38.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:38.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:38.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:38.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:39.030 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:07:39.023+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:39.029295+0000) 2019-10-18T06:07:39.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:39.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:39.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:39.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:40.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:40.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:40.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:40.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:40.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:07:40.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:40.808965+0000) 2019-10-18T06:07:41.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:41.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:41.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:41.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:41.539 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:07:41.535+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:41.538160+0000) 2019-10-18T06:07:41.848 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:41.844+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:41.846983+0000) 2019-10-18T06:07:42.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:42.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:42.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:42.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:43.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:43.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:43.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:43.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:43.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:43.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:44.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:44.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:44.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:44.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:45.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:45.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:45.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:45.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:46.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:46.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:46.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:46.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:47.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:47.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:47.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:47.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:48.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:48.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:48.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:48.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:48.517 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:48.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:49.031 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:07:49.023+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:49.029478+0000) 2019-10-18T06:07:49.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:49.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:49.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:49.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:50.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:50.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:50.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:50.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:50.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:07:50.804+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:50.809064+0000) 2019-10-18T06:07:51.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:51.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:51.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:51.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:51.539 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:07:51.535+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:51.538403+0000) 2019-10-18T06:07:51.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:07:51.844+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:51.847084+0000) 2019-10-18T06:07:52.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:52.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:52.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:52.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:53.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:53.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:53.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:53.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:53.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:53.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:54.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:54.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:54.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:55.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:55.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:55.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:55.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:56.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:56.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:56.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:56.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:57.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:57.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:57.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:57.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:58.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:58.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:58.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:58.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:07:58.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:07:58.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:07:59.031 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:07:59.023+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:07:59.029689+0000) 2019-10-18T06:07:59.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:59.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:07:59.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:07:59.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:00.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:00.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:00.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:00.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:00.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:08:00.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:00.809167+0000) 2019-10-18T06:08:01.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:01.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:01.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:01.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:01.540 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:08:01.535+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:01.538588+0000) 2019-10-18T06:08:01.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:08:01.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:01.847183+0000) 2019-10-18T06:08:02.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:02.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:02.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:02.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:03.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:03.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:03.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:03.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:03.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:03.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:04.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:03.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:04.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:03.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:04.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:04.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:04.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:04.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:06.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:06.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:06.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:06.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:07.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:07.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:08.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:08.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:08.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:08.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:08.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:08.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:09.033 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:08:09.023+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:09.029872+0000) 2019-10-18T06:08:09.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:09.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:09.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:09.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:10.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:10.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:10.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:10.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:10.810 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:08:10.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:10.809267+0000) 2019-10-18T06:08:11.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:11.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:11.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:11.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:11.540 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:08:11.535+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:11.538773+0000) 2019-10-18T06:08:11.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:08:11.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:11.847277+0000) 2019-10-18T06:08:12.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:12.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:12.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:12.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:13.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:13.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:13.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:13.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:13.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:13.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:14.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:14.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:14.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:14.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:15.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:14.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:15.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:15.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:15.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:15.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:15.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:15.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:17.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:17.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:17.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:17.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:18.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:17.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:18.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:17.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:18.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:18.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:18.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:18.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:18.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:18.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:19.031 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:08:19.023+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:19.030014+0000) 2019-10-18T06:08:19.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:19.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:19.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:19.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:20.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:08:20.803+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:20.809365+0000) 2019-10-18T06:08:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:20.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:20.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:21.540 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:08:21.535+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:21.538961+0000) 2019-10-18T06:08:21.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:08:21.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:21.847373+0000) 2019-10-18T06:08:21.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:21.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:21.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:21.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:22.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:22.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:22.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:22.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:23.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:23.519+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:23.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:23.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:23.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:23.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:24.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:24.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:24.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:24.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:25.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:25.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:25.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:25.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:26.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:26.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:26.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:26.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:27.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:27.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:27.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:27.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:28.524 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:28.519+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:28.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:28.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:28.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:28.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:29.031 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:08:29.027+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:29.030222+0000) 2019-10-18T06:08:29.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:29.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:29.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:29.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:30.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:08:30.807+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:30.809469+0000) 2019-10-18T06:08:30.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:30.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:30.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:30.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:31.540 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:08:31.535+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:31.539127+0000) 2019-10-18T06:08:31.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:31.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:31.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:31.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:31.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:08:31.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:31.847474+0000) 2019-10-18T06:08:32.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:32.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:32.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:32.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:33.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:33.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:33.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:33.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:33.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:33.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:34.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:34.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:34.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:34.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:35.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:35.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:35.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:35.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:36.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:36.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:36.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:36.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:37.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:37.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:37.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:37.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:38.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:38.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:38.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:38.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:38.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:38.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:39.031 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:08:39.026+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:39.030373+0000) 2019-10-18T06:08:39.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:39.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:39.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:39.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:40.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:40.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:40.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:40.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:40.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:08:40.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:40.809567+0000) 2019-10-18T06:08:41.540 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:08:41.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:41.539267+0000) 2019-10-18T06:08:41.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:41.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:41.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:41.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:41.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:08:41.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:41.847572+0000) 2019-10-18T06:08:42.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:42.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:42.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:42.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:43.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:43.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:43.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:43.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:43.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:43.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:44.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:44.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:44.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:44.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:45.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:45.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:45.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:45.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:46.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:46.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:46.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:46.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:47.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:47.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:47.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:47.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:48.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:48.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:48.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:48.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:48.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:48.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:49.032 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:08:49.026+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:49.030522+0000) 2019-10-18T06:08:49.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:49.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:49.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:49.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:50.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:08:50.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:50.809662+0000) 2019-10-18T06:08:50.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:50.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:50.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:50.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:51.540 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:08:51.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:51.539424+0000) 2019-10-18T06:08:51.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:51.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:51.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:51.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:51.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:08:51.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:51.847669+0000) 2019-10-18T06:08:52.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:52.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:52.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:52.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:53.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:53.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:53.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:53.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:53.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:53.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:54.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:54.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:54.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:54.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:55.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:55.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:55.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:55.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:56.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:56.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:56.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:56.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:57.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:57.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:57.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:57.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:58.537 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:08:58.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:08:58.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:58.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:58.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:58.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:08:59.032 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:08:59.026+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:08:59.030710+0000) 2019-10-18T06:08:59.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:59.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:08:59.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:08:59.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:00.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:09:00.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:00.809765+0000) 2019-10-18T06:09:00.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:00.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:00.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:00.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:01.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:09:01.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:01.539612+0000) 2019-10-18T06:09:01.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:09:01.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:01.847765+0000) 2019-10-18T06:09:01.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:01.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:01.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:01.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:02.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:02.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:02.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:02.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:03.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:03.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:03.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:03.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:03.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:03.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:04.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:04.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:04.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:04.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:05.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:05.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:05.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:05.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:06.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:06.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:06.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:06.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:07.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:07.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:07.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:07.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:08.530 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:08.521+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:08.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:08.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:08.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:08.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:09.032 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:09:09.026+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:09.030892+0000) 2019-10-18T06:09:09.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:09.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:09.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:09.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:10.813 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:09:10.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:10.809870+0000) 2019-10-18T06:09:10.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:10.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:10.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:10.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:11.541 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:09:11.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:11.539801+0000) 2019-10-18T06:09:11.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:11.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:11.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:11.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:11.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:09:11.841+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:11.847863+0000) 2019-10-18T06:09:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:12.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:12.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:12.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:13.531 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:13.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:13.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:13.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:13.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:13.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:14.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:14.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:14.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:14.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:15.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:15.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:15.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:15.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:16.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:16.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:16.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:16.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:17.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:17.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:17.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:17.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:18.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:18.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:18.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:18.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:18.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:18.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:19.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:09:19.026+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:19.031074+0000) 2019-10-18T06:09:19.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:19.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:19.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:19.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:20.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:09:20.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:20.809966+0000) 2019-10-18T06:09:20.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:20.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:20.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:20.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:21.541 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:09:21.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:21.539944+0000) 2019-10-18T06:09:21.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:09:21.841+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:21.847959+0000) 2019-10-18T06:09:21.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:21.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:21.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:21.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:22.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:22.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:22.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:22.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:23.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:23.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:23.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:23.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:23.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:23.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:24.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:24.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:24.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:24.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:25.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:25.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:25.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:25.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:26.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:26.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:26.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:26.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:27.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:27.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:27.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:27.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:28.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:28.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:28.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:28.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:28.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:28.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:29.032 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:09:29.026+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:29.031258+0000) 2019-10-18T06:09:29.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:29.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:29.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:29.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:30.811 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:09:30.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:30.810068+0000) 2019-10-18T06:09:30.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:30.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:30.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:30.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:31.541 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:09:31.534+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:31.540089+0000) 2019-10-18T06:09:31.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:09:31.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:31.848059+0000) 2019-10-18T06:09:31.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:31.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:31.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:31.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:32.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:32.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:32.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:32.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:33.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:33.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:33.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:33.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:33.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:33.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:34.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:34.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:34.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:34.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:35.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:35.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:35.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:35.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:36.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:36.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:36.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:36.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:37.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:37.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:37.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:37.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:38.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:38.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:38.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:38.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:38.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:38.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:39.032 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:09:39.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:39.031442+0000) 2019-10-18T06:09:39.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:39.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:39.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:39.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:40.812 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:09:40.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:40.810173+0000) 2019-10-18T06:09:40.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:40.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:40.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:40.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:41.541 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:09:41.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:41.540255+0000) 2019-10-18T06:09:41.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:09:41.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:41.848156+0000) 2019-10-18T06:09:41.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:41.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:41.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:41.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:42.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:42.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:42.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:42.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:43.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:43.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:43.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:43.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:43.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:43.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:44.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:44.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:44.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:44.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:45.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:45.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:45.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:45.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:46.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:46.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:46.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:46.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:47.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:47.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:47.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:47.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:48.537 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:48.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:48.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:48.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:48.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:48.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:49.033 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:09:49.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:49.031658+0000) 2019-10-18T06:09:49.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:49.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:49.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:49.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:50.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:50.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:50.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:50.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:50.813 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:09:50.808+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:50.810269+0000) 2019-10-18T06:09:51.542 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:09:51.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:51.540447+0000) 2019-10-18T06:09:51.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:51.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:51.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:51.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:51.849 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:09:51.844+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:51.848256+0000) 2019-10-18T06:09:52.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:52.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:52.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:52.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:53.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:53.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:53.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:53.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:53.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:53.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:54.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:54.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:54.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:54.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:55.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:55.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:55.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:55.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:56.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:56.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:56.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:56.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:57.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:57.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:57.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:57.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:58.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:09:58.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:09:58.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:58.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:58.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:58.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:09:59.033 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:09:59.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:09:59.031845+0000) 2019-10-18T06:09:59.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:59.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:09:59.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:09:59.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:00.812 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:10:00.808+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:00.810372+0000) 2019-10-18T06:10:00.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:00.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:00.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:00.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:01.542 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:10:01.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:01.540631+0000) 2019-10-18T06:10:01.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:10:01.844+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:01.848357+0000) 2019-10-18T06:10:02.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:01.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:02.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:01.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:03.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:03.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:03.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:03.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:03.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:03.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:04.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:04.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:04.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:04.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:05.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:05.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:05.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:05.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:06.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:06.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:06.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:06.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:07.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:07.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:07.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:08.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:08.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:08.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:08.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:08.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:08.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:09.033 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:10:09.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:09.032041+0000) 2019-10-18T06:10:09.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:09.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:09.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:09.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:10.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:10.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:10.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:10.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:10.812 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:10:10.808+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:10.810472+0000) 2019-10-18T06:10:11.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:11.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:11.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:11.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:11.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:10:11.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:11.540817+0000) 2019-10-18T06:10:11.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:10:11.844+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:11.848454+0000) 2019-10-18T06:10:12.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:12.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:12.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:12.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:13.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:13.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:13.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:13.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:13.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:13.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:14.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:14.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:14.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:14.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:15.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:15.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:15.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:15.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:16.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:16.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:16.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:16.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:17.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:17.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:17.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:17.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:18.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:18.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:18.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:18.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:18.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:18.535+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:19.033 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:10:19.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:19.032254+0000) 2019-10-18T06:10:19.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:19.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:19.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:19.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:20.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:20.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:20.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:20.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:20.813 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:10:20.807+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:20.810571+0000) 2019-10-18T06:10:21.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:21.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:21.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:21.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:21.542 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:10:21.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:21.540997+0000) 2019-10-18T06:10:21.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:10:21.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:21.848550+0000) 2019-10-18T06:10:22.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:22.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:22.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:22.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:23.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:23.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:23.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:23.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:23.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:23.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:24.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:24.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:24.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:24.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:25.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:25.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:25.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:25.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:26.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:26.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:26.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:26.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:27.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:27.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:27.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:27.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:28.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:28.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:28.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:28.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:28.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:28.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:29.034 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:10:29.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:29.032470+0000) 2019-10-18T06:10:29.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:29.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:29.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:29.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:30.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:30.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:30.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:30.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:30.812 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:10:30.807+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:30.810672+0000) 2019-10-18T06:10:31.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:31.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:31.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:31.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:31.542 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:10:31.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:31.541151+0000) 2019-10-18T06:10:31.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:10:31.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:31.848650+0000) 2019-10-18T06:10:32.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:32.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:32.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:32.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:33.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:33.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:33.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:33.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:33.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:33.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:34.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:34.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:34.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:34.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:35.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:35.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:35.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:35.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:36.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:36.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:36.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:36.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:37.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:37.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:37.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:37.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:38.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:38.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:38.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:38.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:38.546 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:38.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:39.034 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:10:39.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:39.032680+0000) 2019-10-18T06:10:39.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:39.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:39.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:39.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:40.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:40.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:40.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:40.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:40.812 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:10:40.807+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:40.810769+0000) 2019-10-18T06:10:41.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:41.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:41.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:41.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:41.542 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:10:41.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:41.541296+0000) 2019-10-18T06:10:41.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:10:41.843+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:41.848748+0000) 2019-10-18T06:10:42.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:42.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:42.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:42.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:43.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:43.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:43.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:43.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:43.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:43.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:44.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:44.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:44.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:44.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:45.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:45.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:45.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:45.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:46.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:46.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:46.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:46.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:47.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:47.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:47.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:47.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:48.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:48.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:48.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:48.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:48.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:48.543+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:49.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:10:49.029+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:49.032891+0000) 2019-10-18T06:10:49.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:49.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:49.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:49.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:50.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:50.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:50.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:50.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:50.812 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:10:50.807+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:50.810866+0000) 2019-10-18T06:10:51.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:51.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:51.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:51.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:51.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:10:51.537+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:51.541445+0000) 2019-10-18T06:10:51.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:10:51.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:51.848850+0000) 2019-10-18T06:10:52.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:52.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:52.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:52.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:53.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:53.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:53.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:53.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:53.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:53.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:54.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:54.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:54.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:54.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:55.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:55.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:55.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:56.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:56.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:56.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:56.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:57.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:57.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:57.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:57.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:58.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:58.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:58.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:58.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:10:58.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:10:58.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:10:59.035 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:10:59.028+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:10:59.033105+0000) 2019-10-18T06:10:59.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:59.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:10:59.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:10:59.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:00.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:00.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:00.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:00.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:00.813 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:00.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:00.810960+0000) 2019-10-18T06:11:00.813 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:00.806+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:00.811847+0000) 2019-10-18T06:11:01.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:01.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:01.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:01.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:01.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:11:01.536+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:01.541617+0000) 2019-10-18T06:11:01.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:11:01.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:01.848948+0000) 2019-10-18T06:11:02.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:02.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:02.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:02.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:03.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:03.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:03.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:03.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:03.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:03.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:04.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:04.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:04.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:04.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:05.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:05.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:05.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:05.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:05.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:05.306+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:05.311148+0000) 2019-10-18T06:11:06.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:06.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:06.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:06.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:07.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:07.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:08.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:08.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:08.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:08.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:08.551 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:08.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:09.034 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:11:09.028+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:09.033315+0000) 2019-10-18T06:11:09.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:09.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:09.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:09.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:10.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:10.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:10.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:10.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:11.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:11.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:11.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:11.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:11.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:11:11.536+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:11.541801+0000) 2019-10-18T06:11:11.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:11:11.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:11.849043+0000) 2019-10-18T06:11:12.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:12.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:12.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:12.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:13.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:13.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:13.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:13.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:13.552 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:13.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:14.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:14.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:14.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:14.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:15.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:15.306+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:15.311256+0000) 2019-10-18T06:11:15.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:15.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:15.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:15.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:16.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:16.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:16.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:16.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:17.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:17.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:17.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:18.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:18.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:18.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:18.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:18.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:18.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:19.035 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:11:19.028+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:19.033471+0000) 2019-10-18T06:11:19.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:19.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:19.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:19.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:20.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:20.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:20.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:20.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:21.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:21.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:21.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:21.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:21.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:11:21.536+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:21.541996+0000) 2019-10-18T06:11:21.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:11:21.842+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:21.849144+0000) 2019-10-18T06:11:22.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:22.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:00:44.256092+0000 (oldest deadline 2019-10-18T06:01:04.256092+0000) 2019-10-18T06:11:22.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:22.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:23.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:23.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:23.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:23.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:24.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:24.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:25.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:25.306+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:25.311357+0000) 2019-10-18T06:11:25.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:25.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:26.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:26.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:27.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:27.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:28.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:28.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:28.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:28.545+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:29.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:11:29.028+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:29.033666+0000) 2019-10-18T06:11:29.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:29.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:30.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:30.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:31.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:31.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:31.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:11:31.540+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:31.542204+0000) 2019-10-18T06:11:31.850 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:11:31.841+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:31.849240+0000) 2019-10-18T06:11:32.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:32.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:33.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:33.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:33.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:33.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:34.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:34.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:35.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:35.305+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:35.311459+0000) 2019-10-18T06:11:35.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:35.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:36.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:36.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:37.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:37.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:38.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:38.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:38.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:38.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:39.035 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:11:39.028+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:39.033852+0000) 2019-10-18T06:11:39.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:39.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:40.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:40.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:41.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:41.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:41.543 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:11:41.540+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:41.542394+0000) 2019-10-18T06:11:41.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:11:41.841+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:41.849335+0000) 2019-10-18T06:11:42.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:42.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:43.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:43.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:43.557 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:43.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:44.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:44.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:45.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:45.305+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:45.311559+0000) 2019-10-18T06:11:45.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:45.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:45.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:45.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:46.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:46.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:46.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:46.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:47.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:47.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:47.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:47.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:48.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:48.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:48.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:48.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:48.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:48.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:49.035 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:11:49.028+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:49.034065+0000) 2019-10-18T06:11:49.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:49.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:49.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:49.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:50.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:50.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:50.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:50.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:51.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:51.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:51.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:51.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:51.544 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:11:51.540+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:51.542548+0000) 2019-10-18T06:11:51.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:11:51.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:51.849434+0000) 2019-10-18T06:11:52.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:52.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:52.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:52.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:53.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:53.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:53.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:53.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:53.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:53.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:54.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:54.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:54.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:54.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 492 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:55.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:11:55.309+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:55.311666+0000) 2019-10-18T06:11:55.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:55.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:55.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:55.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:56.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:56.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:56.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:56.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:57.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:57.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:57.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:57.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:58.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:58.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:58.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:58.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:11:58.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:11:58.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:11:59.035 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:11:59.032+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:11:59.034277+0000) 2019-10-18T06:11:59.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:59.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:11:59.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:11:59.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:00.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:00.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:00.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:00.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:01.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:01.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:01.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:01.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:01.544 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:12:01.540+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:01.542719+0000) 2019-10-18T06:12:01.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:12:01.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:01.849540+0000) 2019-10-18T06:12:02.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:02.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:02.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:02.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:03.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:03.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:03.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:03.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:03.560 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:03.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:04.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:04.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:04.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:04.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:05.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:12:05.308+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:05.311759+0000) 2019-10-18T06:12:05.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:05.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:05.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:05.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:06.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:06.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:06.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:06.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:07.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:07.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:07.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:07.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:08.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:08.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:08.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:08.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:08.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:08.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:09.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:12:09.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:09.034467+0000) 2019-10-18T06:12:09.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:09.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:09.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:09.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:10.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:10.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:10.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:10.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:11.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:11.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:11.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:11.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:11.544 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:12:11.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:11.542904+0000) 2019-10-18T06:12:11.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:12:11.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:11.849636+0000) 2019-10-18T06:12:12.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:12.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:12.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:12.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:13.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:13.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:13.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:13.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:13.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:13.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:14.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:14.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:14.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:14.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:15.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:12:15.308+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:15.311858+0000) 2019-10-18T06:12:15.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:15.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:15.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:15.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:16.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:16.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:16.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:16.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:17.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:17.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:17.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:17.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:18.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:18.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:18.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:18.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:18.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:18.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:19.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:12:19.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:19.034676+0000) 2019-10-18T06:12:19.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:19.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:19.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:19.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:20.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:20.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:20.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:20.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:21.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:21.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:21.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:21.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:21.544 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:12:21.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:21.543090+0000) 2019-10-18T06:12:21.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:12:21.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:21.849732+0000) 2019-10-18T06:12:22.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:22.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:22.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:22.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:23.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:23.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:23.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:23.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:23.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:23.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:24.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:24.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:24.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:24.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:25.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:12:25.308+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:25.311961+0000) 2019-10-18T06:12:25.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:25.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:25.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:25.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:26.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:26.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:26.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:26.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:27.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:27.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:27.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:27.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:28.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:28.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:28.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:28.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:28.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:28.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:29.039 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:12:29.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:29.034901+0000) 2019-10-18T06:12:29.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:29.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:29.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:29.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:30.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:30.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:30.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:30.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:31.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:31.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:31.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:31.544 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:12:31.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:31.543253+0000) 2019-10-18T06:12:31.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:12:31.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:31.849835+0000) 2019-10-18T06:12:32.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:32.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:32.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:32.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:33.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:33.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:33.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:33.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:33.567 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:33.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:34.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:34.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:34.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:34.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:35.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:12:35.308+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:35.312059+0000) 2019-10-18T06:12:35.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:35.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:35.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:35.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:36.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:36.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:36.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:36.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:37.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:37.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:37.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:37.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:38.567 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:38.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:38.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:38.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:38.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:38.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:39.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:12:39.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:39.035074+0000) 2019-10-18T06:12:39.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:39.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:39.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:39.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:40.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:40.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:40.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:40.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:41.545 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:12:41.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:41.543421+0000) 2019-10-18T06:12:41.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:41.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:41.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:41.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:41.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:12:41.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:41.849931+0000) 2019-10-18T06:12:42.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:42.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:42.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:42.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:43.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:43.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:43.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:43.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:43.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:43.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:44.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:44.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:44.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:44.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:45.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:12:45.307+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:45.312162+0000) 2019-10-18T06:12:45.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:45.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:45.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:45.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:46.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:46.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:46.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:46.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:47.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:47.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:47.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:47.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:48.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:48.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:48.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:48.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:48.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:48.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:49.036 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:12:49.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:49.035283+0000) 2019-10-18T06:12:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:49.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:49.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:50.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:50.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:50.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:50.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:51.545 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:12:51.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:51.543555+0000) 2019-10-18T06:12:51.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:51.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:51.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:51.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:51.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:12:51.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:51.850033+0000) 2019-10-18T06:12:52.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:52.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:52.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:52.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:53.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:53.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:53.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:53.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:53.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:53.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:54.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:54.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:54.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:54.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:55.314 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:12:55.307+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:55.312265+0000) 2019-10-18T06:12:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:55.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:55.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:56.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:56.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:56.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:56.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:57.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:57.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:57.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:57.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:58.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:12:58.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:12:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:58.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:58.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:12:59.037 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:12:59.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:12:59.035489+0000) 2019-10-18T06:12:59.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:59.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:12:59.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:12:59.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:00.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:00.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:00.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:00.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:01.545 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:13:01.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:01.543741+0000) 2019-10-18T06:13:01.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:01.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:01.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:01.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:01.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:13:01.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:01.850130+0000) 2019-10-18T06:13:02.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:02.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:02.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:02.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:03.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:03.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:03.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:03.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:03.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:03.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:04.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:04.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:04.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:04.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:05.313 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:05.307+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:05.312361+0000) 2019-10-18T06:13:05.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:05.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:05.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:05.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:06.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:06.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:06.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:06.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:07.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:07.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:07.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:07.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:08.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:08.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:08.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:08.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:08.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:08.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:09.037 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:13:09.031+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:09.035675+0000) 2019-10-18T06:13:09.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:09.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:09.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:09.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:10.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:10.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:10.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:10.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:11.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:13:11.539+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:11.543923+0000) 2019-10-18T06:13:11.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:11.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:11.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:11.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:11.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:13:11.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:11.850243+0000) 2019-10-18T06:13:12.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:12.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:12.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:12.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:13.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:13.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:13.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:13.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:13.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:13.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:14.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:14.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:14.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:14.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:15.316 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:15.306+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:15.312454+0000) 2019-10-18T06:13:15.316 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:15.310+0000 7fb64332f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:15.313928+0000) 2019-10-18T06:13:15.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:15.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:15.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:15.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:16.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:16.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:16.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:16.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:17.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:17.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:17.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:17.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:18.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:18.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:18.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:18.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:18.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:18.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:19.037 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:13:19.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:19.035885+0000) 2019-10-18T06:13:19.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:19.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:19.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:19.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:19.814 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:19.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:19.812643+0000) 2019-10-18T06:13:20.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:20.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:20.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:20.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:21.545 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:13:21.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:21.544123+0000) 2019-10-18T06:13:21.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:21.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:21.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:21.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:21.851 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:13:21.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:21.850341+0000) 2019-10-18T06:13:22.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:22.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:22.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:22.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:23.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:23.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:23.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:23.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:23.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:23.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:24.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:24.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:24.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:24.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:25.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:25.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:25.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:25.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:26.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:26.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:26.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:26.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:27.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:27.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:27.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:27.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:28.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:28.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:28.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:28.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:28.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:28.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:29.037 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:13:29.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:29.036100+0000) 2019-10-18T06:13:29.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:29.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:29.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:29.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:29.814 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:29.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:29.812746+0000) 2019-10-18T06:13:30.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:30.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:30.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:30.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:31.545 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:13:31.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:31.544307+0000) 2019-10-18T06:13:31.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:31.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:31.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:31.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:31.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:13:31.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:31.850441+0000) 2019-10-18T06:13:32.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:32.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:32.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:32.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:33.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:33.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:33.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:33.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:33.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:33.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:34.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:34.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:34.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:34.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:35.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:35.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:35.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:35.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:36.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:36.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:36.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:36.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:37.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:37.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:37.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:37.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:38.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:38.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:38.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:38.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:38.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:38.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:39.037 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:13:39.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:39.036291+0000) 2019-10-18T06:13:39.825 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:39.806+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:39.812854+0000) 2019-10-18T06:13:39.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:39.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:39.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:39.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:40.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:40.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:40.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:40.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:41.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:13:41.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:41.544491+0000) 2019-10-18T06:13:41.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:13:41.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:41.850535+0000) 2019-10-18T06:13:41.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:41.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:41.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:41.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:42.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:42.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:42.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:42.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:43.578 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:43.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:43.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:43.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:43.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:43.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:44.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:44.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:44.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:44.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:45.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:45.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:45.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:45.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:46.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:46.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:46.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:46.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:47.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:47.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:47.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:47.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:48.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:48.569+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:48.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:48.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:48.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:48.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:49.037 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:13:49.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:49.036500+0000) 2019-10-18T06:13:49.814 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:49.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:49.812951+0000) 2019-10-18T06:13:49.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:49.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:49.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:49.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:50.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:50.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:50.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:50.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:51.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:13:51.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:51.544680+0000) 2019-10-18T06:13:51.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:13:51.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:51.850638+0000) 2019-10-18T06:13:51.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:51.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:51.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:51.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:52.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:52.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:52.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:52.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:53.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:53.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:53.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:53.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:53.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:53.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:54.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:54.869+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:54.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:54.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:55.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:55.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:55.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:55.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:56.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:56.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:56.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:56.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:57.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:57.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:57.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:57.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:58.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:13:58.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:13:58.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:58.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:58.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:58.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:13:59.038 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:13:59.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:59.036681+0000) 2019-10-18T06:13:59.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:13:59.805+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:13:59.813055+0000) 2019-10-18T06:13:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:59.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:13:59.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:13:59.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:00.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:00.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:00.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:00.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:01.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:14:01.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:01.544861+0000) 2019-10-18T06:14:01.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:14:01.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:01.850742+0000) 2019-10-18T06:14:01.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:01.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:01.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:01.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:02.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:02.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:02.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:02.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:03.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:03.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:03.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:03.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:03.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:03.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:04.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:04.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:04.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:04.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:05.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:05.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:05.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:05.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:06.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:06.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:06.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:06.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:07.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:07.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:07.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:07.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:08.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:08.577+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:08.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:08.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:08.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:08.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:09.038 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:14:09.030+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:09.036912+0000) 2019-10-18T06:14:09.814 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:14:09.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:09.813167+0000) 2019-10-18T06:14:09.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:09.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:09.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:09.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:10.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:10.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:10.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:10.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:11.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:14:11.538+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:11.545042+0000) 2019-10-18T06:14:11.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:14:11.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:11.850840+0000) 2019-10-18T06:14:11.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:11.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:11.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:11.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:12.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:12.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:12.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:12.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:13.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:13.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:13.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:13.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:13.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:13.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:14.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:14.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:14.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:14.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:15.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:15.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:15.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:15.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:16.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:16.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:16.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:16.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:17.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:17.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:17.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:17.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:18.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:18.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:18.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:18.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:18.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:18.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:19.038 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:14:19.034+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:19.037131+0000) 2019-10-18T06:14:19.814 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:14:19.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:19.813266+0000) 2019-10-18T06:14:19.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:19.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:19.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:19.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:20.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:20.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:20.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:20.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:21.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:14:21.542+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:21.545213+0000) 2019-10-18T06:14:21.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:21.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:21.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:21.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:21.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:14:21.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:21.850941+0000) 2019-10-18T06:14:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:22.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:22.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:22.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:23.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:23.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:23.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:23.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:23.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:23.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:24.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:24.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:24.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:24.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:25.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:25.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:25.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:25.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:26.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:26.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:26.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:26.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:27.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:27.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:27.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:27.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:28.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:28.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:28.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:28.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:28.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:28.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:29.043 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:14:29.033+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:29.037357+0000) 2019-10-18T06:14:29.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:14:29.808+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:29.813374+0000) 2019-10-18T06:14:29.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:29.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:29.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:29.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:30.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:30.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:30.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:30.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:31.546 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:14:31.541+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:31.545404+0000) 2019-10-18T06:14:31.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:31.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:31.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:31.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:31.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:14:31.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:31.851050+0000) 2019-10-18T06:14:32.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:32.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:32.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:32.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:33.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:33.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:33.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:33.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:33.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:33.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:34.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:34.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:34.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:34.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:35.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:35.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:35.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:35.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:36.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:36.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:36.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:36.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:37.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:37.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:37.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:37.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:38.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:38.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:38.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:38.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:38.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:38.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:39.039 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:14:39.033+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:39.037570+0000) 2019-10-18T06:14:39.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:39.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:39.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:39.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:39.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:14:39.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:39.813462+0000) 2019-10-18T06:14:40.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:40.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:40.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:40.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:41.547 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:14:41.541+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:41.545590+0000) 2019-10-18T06:14:41.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:41.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:41.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:41.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:41.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:14:41.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:41.851149+0000) 2019-10-18T06:14:42.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:42.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:42.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:42.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:43.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:43.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:43.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:43.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:43.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:43.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:44.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:44.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:44.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:44.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:45.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:45.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:45.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:45.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:46.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:46.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:46.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:46.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:47.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:47.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:47.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:47.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:48.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:48.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:48.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:48.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:48.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:48.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:49.039 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:14:49.033+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:49.037791+0000) 2019-10-18T06:14:49.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:49.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:49.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:49.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:49.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:14:49.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:49.813561+0000) 2019-10-18T06:14:50.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:50.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:50.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:50.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:51.547 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:14:51.541+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:51.545781+0000) 2019-10-18T06:14:51.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:51.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:51.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:51.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:51.852 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:14:51.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:51.851250+0000) 2019-10-18T06:14:52.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:52.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:52.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:52.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:53.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:53.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:53.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:53.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:53.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:53.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:54.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:54.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:54.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:54.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:55.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:55.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:55.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:55.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:56.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:56.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:56.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:56.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:57.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:57.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:57.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:57.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:58.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:14:58.583+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:14:58.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:58.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:58.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:58.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:14:59.039 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:14:59.033+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:59.038004+0000) 2019-10-18T06:14:59.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:14:59.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:14:59.813667+0000) 2019-10-18T06:14:59.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:59.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:14:59.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:14:59.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:00.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:00.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:00.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:00.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:01.547 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:15:01.541+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:01.545969+0000) 2019-10-18T06:15:01.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:01.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:01.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:01.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:01.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:15:01.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:01.851349+0000) 2019-10-18T06:15:02.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:02.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:02.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:02.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:03.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:03.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:03.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:03.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:03.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:03.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:04.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:04.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:04.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:04.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:05.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:05.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:05.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:05.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:06.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:06.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:06.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:06.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:07.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:07.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:07.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:07.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:08.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:08.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:08.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:08.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:08.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:08.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:09.040 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:15:09.037+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:09.038220+0000) 2019-10-18T06:15:09.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:09.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:09.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:09.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:09.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:15:09.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:09.813775+0000) 2019-10-18T06:15:10.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:10.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:10.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:10.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:11.547 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:15:11.545+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:11.546169+0000) 2019-10-18T06:15:11.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:11.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:11.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:11.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:11.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:15:11.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:11.851450+0000) 2019-10-18T06:15:12.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:12.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:12.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:12.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:13.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:13.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:13.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:13.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:13.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:13.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:14.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:14.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:14.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:14.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:15.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:15.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:15.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:15.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:16.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:16.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:16.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:16.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:17.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:17.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:17.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:17.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:18.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:18.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:18.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:18.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:18.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:18.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:19.039 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:15:19.037+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:19.038421+0000) 2019-10-18T06:15:19.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:19.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:19.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:19.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:19.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:15:19.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:19.813874+0000) 2019-10-18T06:15:20.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:20.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:20.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:20.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:21.547 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:15:21.545+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:21.546382+0000) 2019-10-18T06:15:21.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:21.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:21.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:21.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:21.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:15:21.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:21.851554+0000) 2019-10-18T06:15:22.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:22.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:22.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:22.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:23.595 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:23.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:23.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:23.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:23.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:23.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:24.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:24.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:24.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:24.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:25.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:25.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:25.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:25.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:26.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:26.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:26.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:26.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:27.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:27.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:27.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:27.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:28.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:28.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:28.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:28.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:28.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:28.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:29.041 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:15:29.037+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:29.038637+0000) 2019-10-18T06:15:29.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:29.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:29.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:29.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:29.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:15:29.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:29.813981+0000) 2019-10-18T06:15:30.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:30.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:30.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:30.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:31.548 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:15:31.545+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:31.546569+0000) 2019-10-18T06:15:31.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:31.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:31.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:31.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:31.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:15:31.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:31.851656+0000) 2019-10-18T06:15:32.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:32.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:32.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:32.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:33.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:33.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:33.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:33.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:33.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:33.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:34.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:34.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:34.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:34.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:35.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:35.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:35.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:35.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:36.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:36.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:36.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:36.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:37.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:37.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:37.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:37.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:38.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:38.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:38.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:38.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:38.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:38.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:39.044 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:15:39.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:39.038811+0000) 2019-10-18T06:15:39.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:39.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:39.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:39.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:39.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:15:39.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:39.814088+0000) 2019-10-18T06:15:40.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:40.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:40.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:40.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:41.548 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:15:41.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:41.546737+0000) 2019-10-18T06:15:41.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:41.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:41.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:41.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:41.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:15:41.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:41.851750+0000) 2019-10-18T06:15:42.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:42.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:42.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:42.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:43.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:43.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:43.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:43.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:43.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:43.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:44.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:44.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:44.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:44.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:45.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:45.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:45.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:45.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:46.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:46.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:46.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:46.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:47.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:47.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:47.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:47.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:48.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:48.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:48.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:48.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:48.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:48.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:49.040 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:15:49.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:49.038975+0000) 2019-10-18T06:15:49.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:49.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:49.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:49.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:49.815 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:15:49.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:49.814184+0000) 2019-10-18T06:15:50.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:50.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:50.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:50.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:51.548 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:15:51.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:51.546923+0000) 2019-10-18T06:15:51.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:51.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:51.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:51.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:51.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:15:51.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:51.851853+0000) 2019-10-18T06:15:52.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:52.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:52.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:52.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:53.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:53.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:53.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:53.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:53.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:53.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:54.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:54.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:54.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:54.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:55.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:55.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:55.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:55.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:56.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:56.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:56.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:56.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:57.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:57.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:57.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:57.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:58.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:58.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:58.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:58.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:58.601 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:15:58.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:15:59.040 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:15:59.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:59.039133+0000) 2019-10-18T06:15:59.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:59.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:15:59.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:15:59.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:15:59.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:15:59.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:15:59.814292+0000) 2019-10-18T06:16:00.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:00.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:00.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:00.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:01.548 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:16:01.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:01.547115+0000) 2019-10-18T06:16:01.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:01.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:01.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:01.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:01.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:16:01.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:01.851955+0000) 2019-10-18T06:16:02.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:02.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:02.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:02.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:03.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:03.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:03.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:03.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:03.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:03.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:04.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:04.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:04.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:04.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:05.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:05.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:05.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:05.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:06.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:06.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:06.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:06.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:07.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:07.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:07.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:07.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:08.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:08.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:08.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:08.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:08.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:08.593+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:09.040 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:16:09.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:09.039319+0000) 2019-10-18T06:16:09.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:09.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:09.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:09.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:09.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:16:09.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:09.814399+0000) 2019-10-18T06:16:10.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:10.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:10.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:10.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:11.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:11.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:11.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:11.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:11.548 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:16:11.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:11.547258+0000) 2019-10-18T06:16:11.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:16:11.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:11.852056+0000) 2019-10-18T06:16:12.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:12.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:12.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:12.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:13.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:13.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:13.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:13.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:13.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:13.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:14.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:14.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:14.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:14.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:15.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:15.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:15.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:15.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:16.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:16.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:16.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:16.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:17.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:17.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:17.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:17.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:18.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:18.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:18.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:18.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:18.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:18.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:19.040 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:16:19.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:19.039507+0000) 2019-10-18T06:16:19.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:19.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:19.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:19.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:19.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:16:19.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:19.814498+0000) 2019-10-18T06:16:20.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:20.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:20.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:20.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:21.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:21.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:21.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:21.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:21.549 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:16:21.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:21.547442+0000) 2019-10-18T06:16:21.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:16:21.845+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:21.852163+0000) 2019-10-18T06:16:22.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:22.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:22.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:22.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:23.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:23.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:23.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:23.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:23.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:23.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:24.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:24.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:24.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:24.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:25.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:25.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:25.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:25.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:26.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:26.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:26.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:26.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:27.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:27.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:27.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:27.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:28.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:28.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:28.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:28.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:28.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:28.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:29.041 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:16:29.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:29.039703+0000) 2019-10-18T06:16:29.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:29.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:29.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:29.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:29.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:16:29.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:29.814605+0000) 2019-10-18T06:16:30.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:30.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:30.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:30.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:31.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:31.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:31.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:31.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:31.549 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:16:31.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:31.547625+0000) 2019-10-18T06:16:31.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:16:31.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:31.852266+0000) 2019-10-18T06:16:32.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:32.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:32.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:32.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:33.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:33.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:33.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:33.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:33.607 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:33.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:34.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:34.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:34.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:34.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:35.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:35.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:35.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:35.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:36.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:36.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:36.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:36.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:37.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:37.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:37.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:37.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:38.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:38.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:38.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:38.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:38.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:38.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:39.041 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:16:39.036+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:39.039882+0000) 2019-10-18T06:16:39.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:39.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:39.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:39.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:39.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:16:39.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:39.814711+0000) 2019-10-18T06:16:40.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:40.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:40.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:40.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:41.549 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:16:41.544+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:41.547809+0000) 2019-10-18T06:16:41.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:41.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:41.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:41.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:41.853 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:16:41.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:41.852367+0000) 2019-10-18T06:16:42.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:42.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:42.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:42.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:43.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:43.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:43.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:43.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:43.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:43.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:44.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:44.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:44.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:44.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:45.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:45.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:45.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:45.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:46.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:46.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:46.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:46.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:47.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:47.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:47.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:47.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:48.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:48.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:48.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:48.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:48.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:48.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:49.041 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:16:49.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:49.040062+0000) 2019-10-18T06:16:49.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:49.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:49.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:49.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:49.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:16:49.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:49.814816+0000) 2019-10-18T06:16:50.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:50.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:50.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:50.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:51.553 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:16:51.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:51.548005+0000) 2019-10-18T06:16:51.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:51.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:51.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:51.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:51.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:16:51.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:51.852471+0000) 2019-10-18T06:16:52.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:52.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:52.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:52.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:53.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:53.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:53.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:53.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:53.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:53.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:54.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:54.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:54.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:54.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:55.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:55.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:55.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:55.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:56.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:56.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:56.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:56.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:57.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:57.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:57.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:57.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:58.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:16:58.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:16:58.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:58.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:58.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:58.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:59.041 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:16:59.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:59.040257+0000) 2019-10-18T06:16:59.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:59.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:16:59.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:16:59.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:16:59.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:16:59.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:16:59.814921+0000) 2019-10-18T06:17:00.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:00.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:00.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:00.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:01.549 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:17:01.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:01.548190+0000) 2019-10-18T06:17:01.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:01.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:01.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:01.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:01.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:17:01.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:01.852571+0000) 2019-10-18T06:17:02.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:02.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:02.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:02.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:03.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:03.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:03.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:03.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:03.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:03.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:04.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:04.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:04.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:04.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:05.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:05.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:05.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:05.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:06.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:06.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:06.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:06.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:07.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:07.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:07.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:07.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:08.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:08.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:08.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:08.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:08.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:08.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:09.041 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:17:09.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:09.040478+0000) 2019-10-18T06:17:09.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:09.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:09.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:09.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:09.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:17:09.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:09.815029+0000) 2019-10-18T06:17:10.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:10.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:10.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:10.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:11.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:17:11.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:11.548392+0000) 2019-10-18T06:17:11.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:11.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:11.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:11.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:11.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:17:11.848+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:11.852670+0000) 2019-10-18T06:17:12.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:12.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:12.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:12.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:13.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:13.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:13.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:13.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:13.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:13.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:14.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:14.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:14.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:14.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:15.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:15.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:15.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:15.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:16.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:16.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:16.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:16.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:17.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:17.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:17.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:17.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:18.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:18.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:18.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:18.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:18.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:18.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:19.042 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:17:19.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:19.040668+0000) 2019-10-18T06:17:19.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:17:19.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:19.815124+0000) 2019-10-18T06:17:19.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:19.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:19.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:19.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:20.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:20.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:20.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:20.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:21.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:17:21.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:21.548575+0000) 2019-10-18T06:17:21.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:17:21.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:21.852773+0000) 2019-10-18T06:17:21.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:21.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:21.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:21.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:22.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:22.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:22.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:22.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:23.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:23.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:24.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:24.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:24.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:24.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:24.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:24.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:24.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:24.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:25.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:25.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:25.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:25.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:26.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:26.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:26.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:26.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:28.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:27.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:28.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:27.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:28.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:28.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:28.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:28.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:28.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:28.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:29.042 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:17:29.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:29.040868+0000) 2019-10-18T06:17:29.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:17:29.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:29.815225+0000) 2019-10-18T06:17:29.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:29.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:29.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:29.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:30.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:30.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:31.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:30.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:31.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:17:31.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:31.548766+0000) 2019-10-18T06:17:31.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:17:31.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:31.852876+0000) 2019-10-18T06:17:32.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:32.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:32.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:32.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:33.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:32.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:33.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:33.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:33.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:33.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:34.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:34.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:34.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:34.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:35.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:34.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:35.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:35.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:35.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:35.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:36.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:36.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:36.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:36.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:37.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:37.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:37.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:37.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:38.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:38.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:38.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:38.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:38.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:38.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:39.042 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:17:39.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:39.041057+0000) 2019-10-18T06:17:39.816 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:17:39.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:39.815328+0000) 2019-10-18T06:17:39.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:39.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:39.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:39.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:40.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:40.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:40.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:40.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:41.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:17:41.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:41.548953+0000) 2019-10-18T06:17:41.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:17:41.847+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:41.852972+0000) 2019-10-18T06:17:41.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:41.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:41.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:41.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:42.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:42.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:42.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:42.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:43.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:43.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:43.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:43.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:43.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:43.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:44.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:44.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:44.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:44.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:45.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:45.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:45.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:45.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:46.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:46.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:46.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:46.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:47.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:47.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:47.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:47.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:48.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:48.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:48.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:48.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:48.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:48.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:49.042 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:17:49.035+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:49.041240+0000) 2019-10-18T06:17:49.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:17:49.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:49.815428+0000) 2019-10-18T06:17:49.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:49.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:49.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:49.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:50.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:50.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:50.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:50.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:51.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:17:51.543+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:51.549136+0000) 2019-10-18T06:17:51.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:51.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:51.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:51.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:51.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:17:51.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:51.853087+0000) 2019-10-18T06:17:52.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:52.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:52.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:52.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:53.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:53.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:53.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:53.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:53.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:53.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:54.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:54.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:54.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:54.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:55.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:55.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:55.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:55.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:56.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:56.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:56.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:56.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:57.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:57.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:57.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:57.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:58.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:17:58.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:17:58.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:58.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:58.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:58.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:59.042 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:17:59.034+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:59.041428+0000) 2019-10-18T06:17:59.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:59.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:17:59.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:17:59.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:17:59.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:17:59.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:17:59.815540+0000) 2019-10-18T06:18:00.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:00.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:00.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:00.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:01.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:18:01.542+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:01.549330+0000) 2019-10-18T06:18:01.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:01.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:01.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:01.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:01.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:18:01.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:01.853170+0000) 2019-10-18T06:18:02.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:02.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:02.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:02.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:03.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:03.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:03.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:03.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:03.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:03.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:04.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:04.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:04.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:04.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:05.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:05.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:05.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:05.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:06.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:06.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:06.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:06.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:07.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:07.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:07.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:07.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:08.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:08.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:08.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:08.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:08.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:08.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:09.043 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:18:09.034+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:09.041624+0000) 2019-10-18T06:18:09.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:09.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:09.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:09.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:09.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:18:09.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:09.815642+0000) 2019-10-18T06:18:10.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:10.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:10.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:10.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:11.550 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:18:11.542+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:11.549519+0000) 2019-10-18T06:18:11.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:11.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:11.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:11.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:11.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:18:11.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:11.853267+0000) 2019-10-18T06:18:12.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:12.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:12.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:12.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:13.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:13.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:13.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:13.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:13.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:13.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:14.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:14.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:14.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:14.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:15.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:15.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:16.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:16.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:16.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:16.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:17.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:17.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:17.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:17.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:18.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:18.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:18.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:18.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:18.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:18.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:19.043 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:18:19.034+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:19.041819+0000) 2019-10-18T06:18:19.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:19.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:19.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:19.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:19.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:18:19.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:19.815742+0000) 2019-10-18T06:18:20.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:20.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:20.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:20.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:21.551 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:18:21.542+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:21.549716+0000) 2019-10-18T06:18:21.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:21.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:21.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:21.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:21.854 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:18:21.846+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:21.853363+0000) 2019-10-18T06:18:22.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:22.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:22.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:22.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:23.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:23.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:23.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:23.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:23.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:23.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:24.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:24.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:24.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:24.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:25.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:25.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:25.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:25.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:26.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:26.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:26.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:26.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:27.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:27.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:27.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:27.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:28.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:28.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:28.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:28.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:28.626 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:28.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:29.043 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:18:29.034+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:29.042027+0000) 2019-10-18T06:18:29.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:29.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:29.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:29.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:29.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:18:29.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:29.815849+0000) 2019-10-18T06:18:30.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:30.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:30.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:30.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:31.551 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:18:31.542+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:31.549910+0000) 2019-10-18T06:18:31.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:31.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:31.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:31.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:31.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:18:31.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:31.853464+0000) 2019-10-18T06:18:32.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:32.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:32.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:32.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:33.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:33.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:33.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:33.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:33.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:33.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:34.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:34.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:34.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:34.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:35.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:35.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:35.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:35.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:36.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:36.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:36.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:36.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:37.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:37.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:37.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:37.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:38.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:38.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:38.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:38.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:38.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:38.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:39.044 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:18:39.038+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:39.042220+0000) 2019-10-18T06:18:39.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:39.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:39.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:39.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:39.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:18:39.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:39.815955+0000) 2019-10-18T06:18:40.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:40.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:40.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:40.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:41.551 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:18:41.542+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:41.550098+0000) 2019-10-18T06:18:41.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:41.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:41.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:41.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:41.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:18:41.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:41.853562+0000) 2019-10-18T06:18:42.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:42.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:42.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:42.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:43.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:43.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:43.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:43.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:43.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:43.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:44.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:44.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:44.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:44.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:45.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:45.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:45.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:45.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:46.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:46.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:46.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:46.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:47.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:47.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:47.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:47.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:48.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:48.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:48.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:48.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:48.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:48.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:49.043 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:18:49.038+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:49.042405+0000) 2019-10-18T06:18:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:49.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:49.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:49.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:49.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:18:49.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:49.816054+0000) 2019-10-18T06:18:50.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:50.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:50.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:50.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:51.551 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:18:51.546+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:51.550351+0000) 2019-10-18T06:18:51.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:51.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:51.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:51.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:51.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:18:51.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:51.853666+0000) 2019-10-18T06:18:52.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:52.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:52.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:52.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:53.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:53.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:53.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:53.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:53.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:53.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:54.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:54.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:54.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:54.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:55.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:55.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:55.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:55.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:56.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:56.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:56.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:56.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:57.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:57.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:57.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:57.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:58.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:58.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:58.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:58.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:58.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:18:58.629+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:18:59.043 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:18:59.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:59.042620+0000) 2019-10-18T06:18:59.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:59.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:18:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:18:59.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:18:59.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:18:59.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:18:59.816162+0000) 2019-10-18T06:19:00.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:00.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:00.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:00.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:01.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:01.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:01.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:01.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:01.551 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:19:01.550+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:01.550538+0000) 2019-10-18T06:19:01.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:19:01.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:01.853762+0000) 2019-10-18T06:19:02.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:02.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:02.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:02.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:03.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:03.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:03.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:03.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:03.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:03.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:04.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:04.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:04.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:04.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:05.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:05.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:05.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:05.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:06.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:06.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:06.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:06.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:07.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:07.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:07.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:07.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:08.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:08.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:08.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:08.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:08.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:08.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:09.044 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:19:09.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:09.042806+0000) 2019-10-18T06:19:09.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:09.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:09.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:09.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:09.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:19:09.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:09.816275+0000) 2019-10-18T06:19:10.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:10.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:10.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:10.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:11.552 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:19:11.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:11.550731+0000) 2019-10-18T06:19:11.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:11.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:11.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:11.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:11.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:19:11.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:11.853859+0000) 2019-10-18T06:19:12.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:12.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:12.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:12.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:13.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:13.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:13.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:13.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:13.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:13.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:14.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:14.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:14.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:14.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:15.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:15.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:15.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:15.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:16.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:16.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:16.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:16.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:17.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:17.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:17.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:17.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:18.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:18.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:18.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:18.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:18.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:18.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:19.044 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:19:19.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:19.042988+0000) 2019-10-18T06:19:19.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:19.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:19.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:19.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:19.817 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:19:19.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:19.816373+0000) 2019-10-18T06:19:20.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:20.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:20.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:20.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:21.552 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:19:21.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:21.550928+0000) 2019-10-18T06:19:21.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:21.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:21.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:21.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:21.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:19:21.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:21.853966+0000) 2019-10-18T06:19:22.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:22.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:22.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:22.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:23.636 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:23.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:23.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:23.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:23.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:23.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:24.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:24.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:24.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:24.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:25.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:25.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:25.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:25.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:26.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:26.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:26.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:26.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:27.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:27.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:27.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:27.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:28.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:28.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:28.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:28.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:28.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:28.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:29.044 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:19:29.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:29.043178+0000) 2019-10-18T06:19:29.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:29.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:29.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:29.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:29.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:19:29.812+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:29.816482+0000) 2019-10-18T06:19:30.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:30.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:30.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:30.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:31.552 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:19:31.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:31.551121+0000) 2019-10-18T06:19:31.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:31.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:31.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:31.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:31.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:19:31.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:31.854069+0000) 2019-10-18T06:19:32.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:32.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:32.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:32.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:33.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:33.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:33.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:33.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:33.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:33.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:34.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:34.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:34.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:34.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:35.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:35.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:35.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:35.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:36.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:36.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:36.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:36.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:37.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:37.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:37.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:37.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:38.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:38.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:38.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:38.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:38.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:38.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:39.045 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:19:39.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:39.043363+0000) 2019-10-18T06:19:39.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:39.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:39.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:39.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:39.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:19:39.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:39.816592+0000) 2019-10-18T06:19:40.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:40.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:40.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:40.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:41.552 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:19:41.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:41.551312+0000) 2019-10-18T06:19:41.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:41.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:41.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:41.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:41.855 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:19:41.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:41.854164+0000) 2019-10-18T06:19:42.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:42.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:42.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:42.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:43.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:43.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:43.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:43.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:43.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:43.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:44.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:44.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:44.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:44.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:45.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:45.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:45.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:45.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:46.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:46.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:46.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:46.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:47.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:47.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:47.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:47.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:48.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:48.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:48.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:48.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:48.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:48.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:49.044 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:19:49.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:49.043551+0000) 2019-10-18T06:19:49.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:49.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:49.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:49.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:49.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:19:49.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:49.816691+0000) 2019-10-18T06:19:50.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:50.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:50.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:50.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:51.552 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:19:51.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:51.551506+0000) 2019-10-18T06:19:51.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:51.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:51.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:51.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:51.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:19:51.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:51.854259+0000) 2019-10-18T06:19:52.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:52.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:52.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:52.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:53.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:53.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:53.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:53.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:53.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:53.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:54.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:54.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:54.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:54.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:55.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:55.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:55.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:55.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:56.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:56.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:56.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:56.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:57.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:57.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:57.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:57.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:58.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:19:58.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:19:58.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:58.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:58.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:58.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:59.045 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:19:59.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:59.043737+0000) 2019-10-18T06:19:59.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:59.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:19:59.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:19:59.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:19:59.825 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:19:59.811+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:19:59.816797+0000) 2019-10-18T06:20:00.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:00.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:00.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:00.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:01.553 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:20:01.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:01.551694+0000) 2019-10-18T06:20:01.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:01.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:01.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:01.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:01.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:20:01.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:01.854359+0000) 2019-10-18T06:20:02.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:02.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:02.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:02.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:03.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:03.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:03.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:03.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:03.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:03.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:04.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:04.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:04.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:04.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:05.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:05.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:05.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:05.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:06.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:06.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:06.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:06.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:07.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:07.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:07.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:07.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:08.643 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:08.639+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:08.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:08.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:08.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:08.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:09.045 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:20:09.041+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:09.043924+0000) 2019-10-18T06:20:09.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:09.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:09.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:09.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:09.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:20:09.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:09.816906+0000) 2019-10-18T06:20:10.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:10.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:10.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:10.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:11.557 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:20:11.549+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:11.551877+0000) 2019-10-18T06:20:11.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:11.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:11.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:11.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:11.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:20:11.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:11.854460+0000) 2019-10-18T06:20:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:12.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:12.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:13.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:13.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:13.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:13.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:13.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:13.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:14.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:14.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:14.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:14.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:15.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:15.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:15.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:15.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:16.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:16.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:16.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:16.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:17.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:17.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:17.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:17.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:18.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:18.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:18.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:18.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:18.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:18.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:19.047 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:20:19.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:19.044113+0000) 2019-10-18T06:20:19.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:19.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:19.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:19.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:19.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:20:19.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:19.817003+0000) 2019-10-18T06:20:20.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:20.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:20.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:20.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:21.553 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:20:21.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:21.552067+0000) 2019-10-18T06:20:21.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:21.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:21.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:21.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:21.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:20:21.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:21.854558+0000) 2019-10-18T06:20:22.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:22.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:22.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:22.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:23.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:23.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:23.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:23.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:23.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:23.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:24.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:24.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:24.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:24.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:25.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:25.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:25.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:25.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:26.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:26.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:26.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:26.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:27.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:27.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:27.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:27.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:28.647 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:28.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:28.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:28.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:28.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:28.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:29.045 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:20:29.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:29.044336+0000) 2019-10-18T06:20:29.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:20:29.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:29.817109+0000) 2019-10-18T06:20:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:29.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:29.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:29.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:30.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:30.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:30.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:30.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:31.553 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:20:31.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:31.552254+0000) 2019-10-18T06:20:31.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:20:31.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:31.854656+0000) 2019-10-18T06:20:31.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:31.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:31.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:31.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:32.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:32.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:32.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:32.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:33.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:33.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:33.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:33.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:33.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:33.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:34.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:34.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:34.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:34.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:35.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:35.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:35.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:35.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:36.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:36.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:36.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:36.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:37.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:37.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:37.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:37.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:38.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:38.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:38.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:38.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:38.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:38.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:39.045 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:20:39.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:39.044535+0000) 2019-10-18T06:20:39.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:20:39.810+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:39.817217+0000) 2019-10-18T06:20:39.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:39.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:39.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:39.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:40.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:40.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:40.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:40.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:41.553 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:20:41.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:41.552440+0000) 2019-10-18T06:20:41.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:20:41.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:41.854760+0000) 2019-10-18T06:20:41.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:41.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:41.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:41.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:42.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:42.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:42.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:42.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:43.649 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:43.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:43.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:43.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:43.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:43.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:44.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:44.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:44.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:45.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:45.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:45.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:45.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:46.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:46.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:46.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:46.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:47.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:47.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:47.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:47.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:48.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:48.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:48.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:48.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:48.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:48.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:49.046 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:20:49.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:49.044722+0000) 2019-10-18T06:20:49.818 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:20:49.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:49.817313+0000) 2019-10-18T06:20:49.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:49.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:49.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:49.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:50.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:50.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:50.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:50.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:51.554 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:20:51.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:51.552630+0000) 2019-10-18T06:20:51.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:20:51.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:51.854866+0000) 2019-10-18T06:20:51.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:51.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:51.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:51.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:52.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:52.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:52.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:52.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:53.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:53.641+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:53.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:53.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:53.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:53.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:54.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:54.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:54.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:54.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:55.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:55.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:55.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:55.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:56.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:56.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:56.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:56.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:57.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:57.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:57.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:57.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:58.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:20:58.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:20:58.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:58.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:58.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:58.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:20:59.046 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:20:59.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:59.044905+0000) 2019-10-18T06:20:59.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:20:59.809+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:20:59.817420+0000) 2019-10-18T06:20:59.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:59.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:20:59.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:20:59.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:00.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:00.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:00.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:00.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:01.554 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:21:01.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:01.552817+0000) 2019-10-18T06:21:01.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:21:01.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:01.854963+0000) 2019-10-18T06:21:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:02.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:02.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:02.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:02.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:02.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:02.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:02.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:03.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:03.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:03.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:03.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:03.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:03.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:04.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:04.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:04.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:04.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:06.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:05.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:06.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:05.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:07.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:07.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:07.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:07.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:08.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:08.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:08.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:08.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:08.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:08.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:08.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:08.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:08.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:08.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:09.046 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:21:09.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:09.045106+0000) 2019-10-18T06:21:09.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:21:09.817+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:09.817538+0000) 2019-10-18T06:21:09.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:09.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:09.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:09.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:10.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:10.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:10.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:10.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 504 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:11.554 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:21:11.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:11.553001+0000) 2019-10-18T06:21:11.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:21:11.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:11.855066+0000) 2019-10-18T06:21:11.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:11.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:11.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:11.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:12.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:12.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:12.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:12.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:13.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:13.649+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:13.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:13.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:13.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:13.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:14.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:14.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:14.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:14.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:15.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:15.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:15.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:15.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:17.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:17.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:17.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:17.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:18.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:18.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:18.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:18.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:18.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:18.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:19.046 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:21:19.040+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:19.045287+0000) 2019-10-18T06:21:19.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:19.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:19.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:19.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:19.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:21:19.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:19.817640+0000) 2019-10-18T06:21:20.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:20.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:20.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:20.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:21.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:21.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:21.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:21.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:21.554 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:21:21.548+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:21.553186+0000) 2019-10-18T06:21:21.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:21:21.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:21.855163+0000) 2019-10-18T06:21:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:22.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:22.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:23.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:23.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:23.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:23.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:23.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:23.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:24.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:24.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:24.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:24.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:25.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:25.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:25.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:25.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:26.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:26.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:26.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:26.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:27.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:27.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:27.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:27.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:28.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:28.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:28.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:28.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:28.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:28.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:29.046 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:21:29.039+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:29.045498+0000) 2019-10-18T06:21:29.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:29.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:29.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:29.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:29.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:21:29.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:29.817740+0000) 2019-10-18T06:21:30.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:30.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:30.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:30.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:31.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:31.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:31.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:31.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:31.555 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:21:31.547+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:31.553371+0000) 2019-10-18T06:21:31.856 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:21:31.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:31.855260+0000) 2019-10-18T06:21:32.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:32.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:32.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:32.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:33.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:33.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:33.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:33.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:33.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:33.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:34.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:34.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:34.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:34.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:35.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:35.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:35.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:35.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:36.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:36.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:36.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:36.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:37.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:37.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:37.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:37.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:38.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:38.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:38.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:38.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:38.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:38.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:39.052 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:21:39.039+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:39.045676+0000) 2019-10-18T06:21:39.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:39.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:39.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:39.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:39.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:21:39.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:39.817845+0000) 2019-10-18T06:21:40.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:40.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:40.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:40.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:41.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:41.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:41.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:41.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:41.554 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:21:41.547+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:41.553549+0000) 2019-10-18T06:21:41.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:21:41.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:41.855359+0000) 2019-10-18T06:21:42.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:42.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:42.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:42.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:43.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:43.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:43.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:43.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:43.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:43.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:44.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:44.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:44.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:44.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:45.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:45.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:45.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:45.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:46.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:46.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:46.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:46.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:47.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:47.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:47.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:47.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:48.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:48.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:48.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:48.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:48.660 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:48.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:49.047 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:21:49.039+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:49.045866+0000) 2019-10-18T06:21:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:49.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:49.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:49.825 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:21:49.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:49.817942+0000) 2019-10-18T06:21:50.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:50.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:50.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:50.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:51.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:51.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:51.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:51.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:51.555 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:21:51.547+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:51.553752+0000) 2019-10-18T06:21:51.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:21:51.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:51.855461+0000) 2019-10-18T06:21:52.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:52.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:52.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:52.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:53.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:53.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:53.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:53.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:53.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:53.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:54.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:54.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:54.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:54.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:55.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:55.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:55.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:55.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:56.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:56.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:11:24.488274+0000 (oldest deadline 2019-10-18T06:11:44.488274+0000) 2019-10-18T06:21:56.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:56.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:57.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:57.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:58.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:58.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:58.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:21:58.655+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:21:59.047 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:21:59.039+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:59.046038+0000) 2019-10-18T06:21:59.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:21:59.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:21:59.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:21:59.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:21:59.818045+0000) 2019-10-18T06:22:00.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:00.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:01.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:01.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:01.555 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:22:01.547+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:01.553930+0000) 2019-10-18T06:22:01.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:22:01.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:01.855557+0000) 2019-10-18T06:22:02.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:02.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:03.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:03.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:03.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:03.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:04.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:04.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:05.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:05.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:06.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:06.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:07.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:07.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:08.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:08.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:08.663 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:08.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:09.047 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:22:09.043+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:09.046233+0000) 2019-10-18T06:22:09.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:09.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:09.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:22:09.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:09.818145+0000) 2019-10-18T06:22:10.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:10.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:11.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:11.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:11.555 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:22:11.547+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:11.554116+0000) 2019-10-18T06:22:11.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:22:11.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:11.855657+0000) 2019-10-18T06:22:12.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:12.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:13.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:13.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:13.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:13.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:14.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:14.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:15.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:15.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:16.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:16.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:17.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:17.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:18.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:18.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:18.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:18.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:18.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:18.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:19.047 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:22:19.043+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:19.046411+0000) 2019-10-18T06:22:19.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:19.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:19.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:19.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:19.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:22:19.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:19.818244+0000) 2019-10-18T06:22:20.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:20.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:20.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:20.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:21.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:21.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:21.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:21.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:21.555 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:22:21.551+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:21.554310+0000) 2019-10-18T06:22:21.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:22:21.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:21.855756+0000) 2019-10-18T06:22:22.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:22.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:22.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:22.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:23.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:23.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:23.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:23.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:23.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:23.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:24.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:24.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:24.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:24.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:25.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:25.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:25.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:25.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:26.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:26.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:26.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:26.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:27.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:27.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:27.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:27.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:28.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:28.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:28.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:28.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:28.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:28.663+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:29.047 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:22:29.043+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:29.046590+0000) 2019-10-18T06:22:29.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:29.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:29.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:29.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:29.819 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:22:29.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:29.818348+0000) 2019-10-18T06:22:30.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:30.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:30.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:30.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:31.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:31.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:31.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:31.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:31.555 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:22:31.551+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:31.554496+0000) 2019-10-18T06:22:31.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:22:31.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:31.855869+0000) 2019-10-18T06:22:32.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:32.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:32.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:32.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:33.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:33.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:33.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:33.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:33.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:33.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:34.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:34.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:34.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:34.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:35.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:35.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:35.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:35.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:36.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:36.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:36.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:36.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:37.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:37.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:37.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:37.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:38.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:38.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:38.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:38.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:38.669 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:38.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:39.048 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:22:39.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:39.046759+0000) 2019-10-18T06:22:39.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:39.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:39.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:39.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:39.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:22:39.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:39.818453+0000) 2019-10-18T06:22:40.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:40.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:40.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:40.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:41.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:41.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:41.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:41.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:41.556 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:22:41.550+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:41.554686+0000) 2019-10-18T06:22:41.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:22:41.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:41.855969+0000) 2019-10-18T06:22:42.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:42.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:42.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:42.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:43.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:43.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:43.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:43.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:43.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:43.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:44.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:44.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:44.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:44.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:45.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:45.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:45.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:45.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:46.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:46.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:46.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:46.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:47.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:47.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:47.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:47.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:48.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:48.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:48.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:48.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:48.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:48.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:49.048 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:22:49.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:49.046935+0000) 2019-10-18T06:22:49.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:49.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:49.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:49.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:49.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:22:49.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:49.818553+0000) 2019-10-18T06:22:50.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:50.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:50.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:50.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:51.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:51.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:51.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:51.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:51.556 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:22:51.550+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:51.554901+0000) 2019-10-18T06:22:51.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:22:51.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:51.856073+0000) 2019-10-18T06:22:52.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:52.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:52.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:52.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:53.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:53.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:53.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:53.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:53.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:53.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:54.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:54.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:54.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:54.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:55.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:55.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:55.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:55.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:56.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:56.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:56.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:56.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:57.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:57.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:57.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:57.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:58.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:58.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:58.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:58.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:58.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:22:58.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:22:59.048 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:22:59.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:59.047161+0000) 2019-10-18T06:22:59.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:59.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:22:59.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:22:59.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:22:59.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:22:59.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:22:59.818658+0000) 2019-10-18T06:23:00.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:00.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:00.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:00.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:01.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:01.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:01.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:01.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:01.556 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:23:01.550+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:01.555119+0000) 2019-10-18T06:23:01.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:23:01.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:01.856171+0000) 2019-10-18T06:23:02.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:02.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:02.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:02.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:03.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:03.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:03.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:03.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:03.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:03.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:04.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:04.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:04.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:04.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:05.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:05.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:05.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:05.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:06.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:06.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:06.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:07.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:07.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:07.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:07.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:08.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:08.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:08.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:08.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:08.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:08.665+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:09.048 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:23:09.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:09.047340+0000) 2019-10-18T06:23:09.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:09.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:09.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:09.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:09.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:23:09.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:09.818766+0000) 2019-10-18T06:23:10.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:10.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:10.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:10.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:11.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:11.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:11.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:11.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:11.556 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:23:11.550+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:11.555331+0000) 2019-10-18T06:23:11.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:23:11.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:11.856274+0000) 2019-10-18T06:23:12.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:12.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:12.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:12.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:13.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:13.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:13.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:13.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:13.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:13.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:14.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:14.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:14.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:14.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:15.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:15.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:15.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:15.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:16.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:16.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:16.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:16.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:17.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:17.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:17.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:17.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:18.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:18.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:18.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:18.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:18.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:18.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:19.048 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:23:19.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:19.047530+0000) 2019-10-18T06:23:19.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:19.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:19.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:19.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:19.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:23:19.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:19.818870+0000) 2019-10-18T06:23:20.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:20.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:20.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:20.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:21.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:21.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:21.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:21.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:21.558 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:23:21.550+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:21.555521+0000) 2019-10-18T06:23:21.857 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:23:21.849+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:21.856379+0000) 2019-10-18T06:23:22.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:22.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:22.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:22.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:23.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:23.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:23.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:23.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:23.677 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:23.673+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:24.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:24.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:24.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:24.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:25.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:25.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:25.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:25.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:26.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:26.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:26.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:26.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:27.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:27.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:27.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:27.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:28.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:28.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:28.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:28.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:28.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:28.673+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:29.049 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:23:29.042+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:29.047747+0000) 2019-10-18T06:23:29.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:29.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:29.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:29.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:29.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:23:29.817+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:29.818976+0000) 2019-10-18T06:23:30.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:30.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:30.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:30.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:31.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:31.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:31.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:31.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:31.557 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:23:31.554+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:31.555712+0000) 2019-10-18T06:23:31.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:23:31.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:31.856478+0000) 2019-10-18T06:23:32.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:32.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:32.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:32.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:33.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:33.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:33.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:33.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:33.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:33.673+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:34.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:34.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:34.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:34.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:35.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:35.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:35.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:35.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:36.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:36.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:36.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:36.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:37.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:37.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:37.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:37.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:38.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:38.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:38.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:38.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:38.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:38.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:39.049 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:23:39.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:39.047930+0000) 2019-10-18T06:23:39.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:39.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:39.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:39.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:39.826 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:23:39.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:39.819083+0000) 2019-10-18T06:23:40.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:40.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:40.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:40.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:41.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:41.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:41.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:41.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:41.557 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:23:41.554+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:41.555900+0000) 2019-10-18T06:23:41.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:23:41.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:41.856576+0000) 2019-10-18T06:23:42.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:42.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:42.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:42.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:43.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:43.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:43.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:43.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:43.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:43.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:44.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:44.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:44.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:44.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:45.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:45.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:45.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:45.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:46.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:46.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:46.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:46.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:47.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:47.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:47.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:47.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:48.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:48.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:48.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:48.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:48.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:48.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:49.049 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:23:49.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:49.048121+0000) 2019-10-18T06:23:49.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:49.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:49.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:49.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:49.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:23:49.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:49.819186+0000) 2019-10-18T06:23:50.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:50.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:50.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:50.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:51.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:51.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:51.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:51.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:51.557 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:23:51.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:51.556120+0000) 2019-10-18T06:23:51.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:23:51.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:51.856677+0000) 2019-10-18T06:23:52.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:52.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:52.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:52.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:53.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:53.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:53.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:53.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:53.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:53.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:54.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:54.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:54.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:54.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:55.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:55.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:55.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:55.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:56.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:56.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:56.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:56.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:57.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:57.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:57.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:57.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:58.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:58.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:58.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:58.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:58.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:23:58.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:23:59.049 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:23:59.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:59.048315+0000) 2019-10-18T06:23:59.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:59.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:23:59.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:23:59.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:23:59.820 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:23:59.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:23:59.819295+0000) 2019-10-18T06:24:00.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:00.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:00.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:00.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:01.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:01.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:01.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:01.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:01.557 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:24:01.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:01.556304+0000) 2019-10-18T06:24:01.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:24:01.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:01.856774+0000) 2019-10-18T06:24:02.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:02.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:02.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:02.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:03.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:03.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:03.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:03.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:03.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:03.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:04.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:04.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:04.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:04.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:05.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:05.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:05.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:05.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:06.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:06.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:06.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:06.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:07.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:07.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:07.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:07.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:08.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:08.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:08.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:08.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:08.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:08.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:09.049 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:24:09.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:09.048499+0000) 2019-10-18T06:24:09.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:09.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:09.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:09.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:09.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:24:09.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:09.819403+0000) 2019-10-18T06:24:10.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:10.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:10.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:10.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:11.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:11.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:11.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:11.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:11.557 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:24:11.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:11.556491+0000) 2019-10-18T06:24:11.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:24:11.852+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:11.856872+0000) 2019-10-18T06:24:12.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:12.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:12.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:12.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:13.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:13.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:13.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:13.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:13.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:13.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:14.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:14.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:14.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:14.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:15.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:15.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:15.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:15.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:16.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:16.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:16.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:16.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:17.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:17.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:17.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:17.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:18.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:18.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:18.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:18.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:18.679+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:19.050 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:24:19.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:19.048684+0000) 2019-10-18T06:24:19.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:19.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:19.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:19.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:19.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:24:19.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:19.819507+0000) 2019-10-18T06:24:20.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:20.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:20.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:20.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:21.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:21.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:21.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:21.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:21.558 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:24:21.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:21.556685+0000) 2019-10-18T06:24:21.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:24:21.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:21.856973+0000) 2019-10-18T06:24:22.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:22.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:22.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:22.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:23.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:23.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:23.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:23.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:23.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:23.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:24.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:24.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:24.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:24.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:25.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:25.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:25.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:25.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:26.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:26.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:26.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:26.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:27.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:27.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:27.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:27.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:28.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:28.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:28.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:28.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:28.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:28.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:29.050 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:24:29.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:29.048880+0000) 2019-10-18T06:24:29.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:29.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:29.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:29.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:29.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:24:29.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:29.819617+0000) 2019-10-18T06:24:30.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:30.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:30.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:30.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:31.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:31.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:31.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:31.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:31.558 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:24:31.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:31.556898+0000) 2019-10-18T06:24:31.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:24:31.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:31.857069+0000) 2019-10-18T06:24:32.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:32.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:32.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:32.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:33.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:33.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:33.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:33.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:33.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:33.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:34.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:34.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:34.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:34.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:35.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:35.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:35.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:35.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:36.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:36.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:36.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:36.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:37.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:37.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:37.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:37.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:38.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:38.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:38.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:38.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:38.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:38.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:39.050 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:24:39.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:39.049064+0000) 2019-10-18T06:24:39.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:39.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:39.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:39.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:39.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:24:39.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:39.819720+0000) 2019-10-18T06:24:40.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:40.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:40.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:40.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:41.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:41.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:41.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:41.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:41.558 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:24:41.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:41.557088+0000) 2019-10-18T06:24:41.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:24:41.851+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:41.857168+0000) 2019-10-18T06:24:42.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:42.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:42.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:42.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:43.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:43.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:43.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:43.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:43.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:43.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:44.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:44.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:44.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:44.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:45.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:45.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:46.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:46.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:46.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:46.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:47.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:47.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:47.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:47.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:48.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:48.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:48.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:48.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:48.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:48.687+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:49.050 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:24:49.045+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:49.049288+0000) 2019-10-18T06:24:49.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:49.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:49.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:49.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:49.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:24:49.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:49.819824+0000) 2019-10-18T06:24:50.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:50.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:50.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:50.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:51.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:51.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:51.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:51.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:51.558 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:24:51.553+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:51.557281+0000) 2019-10-18T06:24:51.858 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:24:51.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:51.857266+0000) 2019-10-18T06:24:52.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:52.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:52.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:52.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:53.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:53.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:53.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:53.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:53.692 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:53.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:54.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:54.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:54.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:55.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:55.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:55.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:55.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:56.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:56.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:56.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:56.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:57.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:57.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:57.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:57.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:58.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:58.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:58.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:58.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:58.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:24:58.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:24:59.050 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:24:59.044+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:59.049477+0000) 2019-10-18T06:24:59.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:59.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:24:59.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:24:59.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:24:59.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:24:59.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:24:59.819932+0000) 2019-10-18T06:25:00.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:00.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:00.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:00.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:01.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:01.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:01.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:01.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:01.558 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:01.552+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:01.557474+0000) 2019-10-18T06:25:01.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:25:01.850+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:01.857371+0000) 2019-10-18T06:25:02.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:02.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:02.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:02.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:03.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:03.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:03.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:03.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:03.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:03.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:04.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:04.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:04.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:04.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:05.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:05.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:05.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:05.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:06.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:06.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:06.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:06.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:07.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:07.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:07.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:07.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:08.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:08.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:08.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:08.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:08.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:08.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:09.051 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:25:09.044+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:09.049679+0000) 2019-10-18T06:25:09.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:09.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:09.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:09.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:09.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:25:09.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:09.820042+0000) 2019-10-18T06:25:10.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:10.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:10.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:10.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:11.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:11.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:11.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:11.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:11.559 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:11.552+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:11.557680+0000) 2019-10-18T06:25:11.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:25:11.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:11.857463+0000) 2019-10-18T06:25:12.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:12.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:12.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:12.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:13.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:13.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:13.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:13.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:13.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:13.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:14.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:14.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:14.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:14.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:15.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:15.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:15.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:15.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:16.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:16.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:16.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:16.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:17.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:17.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:17.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:17.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:18.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:18.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:18.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:18.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:18.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:18.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:19.051 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:25:19.044+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:19.049881+0000) 2019-10-18T06:25:19.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:19.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:19.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:19.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:19.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:25:19.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:19.820145+0000) 2019-10-18T06:25:20.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:20.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:20.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:20.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:21.559 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:21.552+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:21.557883+0000) 2019-10-18T06:25:21.560 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:21.556+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:21.559141+0000) 2019-10-18T06:25:21.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:21.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:21.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:21.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:21.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:25:21.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:21.857560+0000) 2019-10-18T06:25:22.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:22.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:22.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:22.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:23.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:23.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:23.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:23.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:23.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:23.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:24.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:24.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:24.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:24.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:25.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:25.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:25.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:25.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:26.059 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:26.056+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:26.058395+0000) 2019-10-18T06:25:26.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:26.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:26.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:26.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:27.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:27.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:27.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:27.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:28.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:28.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:28.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:28.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:28.698 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:28.689+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:29.051 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:25:29.044+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:29.050071+0000) 2019-10-18T06:25:29.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:29.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:29.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:29.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:29.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:25:29.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:29.820256+0000) 2019-10-18T06:25:30.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:30.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:30.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:30.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:31.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:31.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:31.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:31.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:31.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:25:31.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:31.857656+0000) 2019-10-18T06:25:32.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:32.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:32.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:32.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:33.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:33.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:33.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:33.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:33.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:33.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:34.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:34.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:34.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:34.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:35.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:35.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:35.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:35.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:36.059 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:36.056+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:36.058561+0000) 2019-10-18T06:25:36.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:36.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:36.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:36.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:37.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:37.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:37.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:37.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:38.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:38.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:38.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:38.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:38.700 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:38.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:39.051 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:25:39.048+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:39.050265+0000) 2019-10-18T06:25:39.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:39.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:39.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:39.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:39.821 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:25:39.813+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:39.820360+0000) 2019-10-18T06:25:40.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:40.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:40.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:40.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:41.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:41.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:41.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:41.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:41.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:25:41.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:41.857751+0000) 2019-10-18T06:25:42.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:42.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:42.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:42.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:43.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:43.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:43.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:43.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:43.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:43.697+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:44.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:44.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:44.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:44.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:45.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:45.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:45.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:45.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:46.060 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:46.056+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:46.058754+0000) 2019-10-18T06:25:46.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:46.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:46.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:46.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:47.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:47.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:47.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:47.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:48.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:48.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:48.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:48.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:48.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:48.697+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:49.051 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:25:49.048+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:49.050455+0000) 2019-10-18T06:25:49.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:49.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:49.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:49.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:49.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:25:49.817+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:49.820465+0000) 2019-10-18T06:25:50.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:50.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:50.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:50.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:51.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:51.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:51.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:51.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:51.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:25:51.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:51.857852+0000) 2019-10-18T06:25:52.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:52.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:52.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:52.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:53.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:53.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:53.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:53.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:53.703 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:53.697+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:54.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:54.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:54.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:54.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:55.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:55.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:55.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:55.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:56.061 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:25:56.056+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:56.058936+0000) 2019-10-18T06:25:56.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:56.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:56.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:56.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:57.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:57.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:57.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:57.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:58.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:58.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:58.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:25:58.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:25:59.052 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:25:59.048+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:59.050641+0000) 2019-10-18T06:25:59.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:59.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:25:59.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:25:59.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:25:59.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:25:59.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:25:59.820574+0000) 2019-10-18T06:26:00.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:00.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:00.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:00.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:01.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:01.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:01.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:01.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:01.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:26:01.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:01.857947+0000) 2019-10-18T06:26:02.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:02.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:02.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:02.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:03.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:03.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:03.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:03.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:03.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:03.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:04.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:04.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:04.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:05.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:05.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:05.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:05.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:06.060 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:26:06.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:06.059135+0000) 2019-10-18T06:26:06.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:06.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:06.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:06.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:07.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:07.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:07.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:07.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:08.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:08.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:08.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:08.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:08.705 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:08.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:09.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:26:09.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:09.050831+0000) 2019-10-18T06:26:09.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:09.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:09.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:09.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:09.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:26:09.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:09.820681+0000) 2019-10-18T06:26:10.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:10.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:10.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:10.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:11.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:11.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:11.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:11.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:11.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:26:11.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:11.858053+0000) 2019-10-18T06:26:12.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:12.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:12.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:12.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:13.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:13.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:13.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:13.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:13.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:13.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:14.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:14.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:14.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:14.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:15.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:15.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:15.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:15.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:16.060 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:26:16.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:16.059321+0000) 2019-10-18T06:26:16.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:16.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:16.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:16.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:17.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:17.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:17.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:17.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:18.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:18.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:18.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:18.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:18.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:18.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:19.052 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:26:19.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:19.051019+0000) 2019-10-18T06:26:19.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:19.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:19.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:19.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:19.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:26:19.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:19.820786+0000) 2019-10-18T06:26:20.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:20.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:20.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:20.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:21.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:21.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:21.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:21.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:21.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:26:21.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:21.858151+0000) 2019-10-18T06:26:22.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:22.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:22.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:22.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:23.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:23.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:23.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:23.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:23.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:23.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:24.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:24.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:24.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:24.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:25.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:25.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:25.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:25.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:26.060 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:26:26.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:26.059502+0000) 2019-10-18T06:26:26.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:26.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:26.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:26.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:27.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:27.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:27.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:27.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:28.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:28.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:28.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:28.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:28.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:28.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:29.052 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:26:29.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:29.051213+0000) 2019-10-18T06:26:29.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:29.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:29.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:29.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:29.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:26:29.816+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:29.820910+0000) 2019-10-18T06:26:30.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:30.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:30.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:30.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:31.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:31.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:31.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:31.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:31.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:26:31.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:31.858251+0000) 2019-10-18T06:26:32.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:32.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:32.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:32.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:33.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:33.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:33.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:33.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:33.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:33.703+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:34.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:34.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:34.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:34.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:35.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:35.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:35.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:35.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:36.061 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:26:36.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:36.059695+0000) 2019-10-18T06:26:36.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:36.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:36.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:36.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:37.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:37.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:37.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:37.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:38.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:38.703+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:38.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:38.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:38.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:38.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:39.052 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:26:39.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:39.051391+0000) 2019-10-18T06:26:39.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:39.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:39.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:39.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:39.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:26:39.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:39.821018+0000) 2019-10-18T06:26:40.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:40.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:40.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:40.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:41.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:41.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:41.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:41.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:41.859 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:26:41.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:41.858351+0000) 2019-10-18T06:26:42.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:42.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:42.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:42.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:43.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:43.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:43.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:43.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:43.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:43.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:44.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:44.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:44.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:44.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:45.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:45.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:45.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:45.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:46.061 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:26:46.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:46.059884+0000) 2019-10-18T06:26:46.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:46.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:46.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:46.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:47.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:47.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:47.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:47.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:48.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:48.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:48.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:48.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:48.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:48.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:49.052 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:26:49.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:49.051598+0000) 2019-10-18T06:26:49.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:49.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:49.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:49.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:49.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:26:49.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:49.821122+0000) 2019-10-18T06:26:50.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:50.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:50.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:50.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:51.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:51.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:51.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:51.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:51.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:26:51.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:51.858453+0000) 2019-10-18T06:26:52.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:52.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:52.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:52.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:53.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:53.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:53.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:53.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:53.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:53.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:54.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:54.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:54.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:54.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:55.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:55.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:55.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:55.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:56.061 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:26:56.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:56.060095+0000) 2019-10-18T06:26:56.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:56.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:56.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:56.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:57.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:57.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:57.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:57.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:58.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:58.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:58.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:58.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:58.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:26:58.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:26:59.053 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:26:59.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:59.051780+0000) 2019-10-18T06:26:59.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:59.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:26:59.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:26:59.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:26:59.822 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:26:59.815+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:26:59.821234+0000) 2019-10-18T06:27:00.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:00.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:00.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:00.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:01.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:01.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:01.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:01.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:01.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:27:01.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:01.858548+0000) 2019-10-18T06:27:02.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:02.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:02.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:02.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:03.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:03.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:03.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:03.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:03.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:03.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:04.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:04.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:04.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:04.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:05.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:05.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:05.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:05.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:06.062 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:27:06.055+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:06.060278+0000) 2019-10-18T06:27:06.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:06.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:06.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:06.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:07.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:07.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:07.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:07.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:08.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:08.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:08.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:08.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:08.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:08.710+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:09.053 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:27:09.047+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:09.051975+0000) 2019-10-18T06:27:09.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:09.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:09.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:09.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:09.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:27:09.814+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:09.821344+0000) 2019-10-18T06:27:10.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:10.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:10.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:10.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:11.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:11.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:11.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:11.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:11.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:27:11.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:11.858660+0000) 2019-10-18T06:27:12.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:12.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:12.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:12.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:13.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:13.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:13.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:13.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:13.716 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:13.710+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:14.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:14.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:14.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:14.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:15.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:15.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:15.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:15.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:16.061 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:27:16.054+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:16.060465+0000) 2019-10-18T06:27:16.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:16.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:16.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:16.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:17.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:17.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:17.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:17.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:18.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:18.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:18.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:18.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:18.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:18.710+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:19.053 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:27:19.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:19.052170+0000) 2019-10-18T06:27:19.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:19.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:19.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:19.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:19.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:27:19.818+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:19.821458+0000) 2019-10-18T06:27:20.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:20.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:20.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:20.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:21.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:21.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:21.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:21.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:21.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:27:21.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:21.858761+0000) 2019-10-18T06:27:22.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:22.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:22.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:22.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:23.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:23.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:23.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:23.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:23.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:23.710+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:24.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:24.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:24.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:24.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:25.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:25.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:25.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:25.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:26.062 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:27:26.054+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:26.060615+0000) 2019-10-18T06:27:26.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:26.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:26.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:26.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:27.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:27.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:27.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:27.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:28.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:28.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:28.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:28.630+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:28.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:28.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:29.055 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:27:29.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:29.052371+0000) 2019-10-18T06:27:29.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:29.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:29.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:29.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:29.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:27:29.818+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:29.821575+0000) 2019-10-18T06:27:30.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:30.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:30.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:30.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:31.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:31.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:31.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:31.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:31.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:27:31.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:31.858859+0000) 2019-10-18T06:27:32.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:32.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:32.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:32.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:33.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:33.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:33.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:33.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:33.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:33.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:34.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:34.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:34.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:34.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:35.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:35.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:35.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:35.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:36.062 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:27:36.054+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:36.060803+0000) 2019-10-18T06:27:36.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:36.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:36.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:36.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:37.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:37.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:37.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:37.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:38.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:38.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:38.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:38.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:38.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:38.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:39.053 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:27:39.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:39.052559+0000) 2019-10-18T06:27:39.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:39.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:39.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:39.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:39.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:27:39.818+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:39.821684+0000) 2019-10-18T06:27:40.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:40.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:40.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:40.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:41.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:41.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:41.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:41.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:41.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:27:41.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:41.858958+0000) 2019-10-18T06:27:42.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:42.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:42.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:42.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:43.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:43.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:43.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:43.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:43.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:43.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:44.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:44.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:45.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:45.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:45.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:45.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:46.062 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:27:46.054+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:46.061003+0000) 2019-10-18T06:27:46.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:46.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:46.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:46.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:47.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:47.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:47.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:47.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:48.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:48.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:48.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:48.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:48.723 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:48.713+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:49.054 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:27:49.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:49.052737+0000) 2019-10-18T06:27:49.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:49.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:49.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:49.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:49.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:27:49.817+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:49.821786+0000) 2019-10-18T06:27:50.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:50.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:50.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:50.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:51.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:51.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:51.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:51.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:51.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:27:51.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:51.859061+0000) 2019-10-18T06:27:52.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:52.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:52.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:52.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:53.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:53.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:53.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:53.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:53.724 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:53.717+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:54.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:54.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:54.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:54.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:55.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:55.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:55.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:55.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:56.062 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:27:56.054+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:56.061172+0000) 2019-10-18T06:27:56.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:56.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:56.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:56.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:57.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:57.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:57.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:57.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:58.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:58.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:58.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:58.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:58.724 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:27:58.717+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:27:59.054 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:27:59.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:59.052920+0000) 2019-10-18T06:27:59.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:59.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:27:59.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:27:59.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:27:59.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:27:59.817+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:27:59.821892+0000) 2019-10-18T06:28:00.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:00.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:00.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:00.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:01.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:01.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:01.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:01.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:01.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:28:01.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:01.859159+0000) 2019-10-18T06:28:02.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:02.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:02.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:02.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:03.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:03.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:03.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:03.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:03.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:03.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:04.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:04.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:04.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:04.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:05.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:05.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:05.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:05.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:06.062 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:28:06.054+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:06.061365+0000) 2019-10-18T06:28:06.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:06.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:06.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:06.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:07.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:07.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:07.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:07.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:08.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:08.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:08.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:08.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:08.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:08.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:09.054 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:28:09.046+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:09.053143+0000) 2019-10-18T06:28:09.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:09.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:09.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:09.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:09.825 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:28:09.821+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:09.821996+0000) 2019-10-18T06:28:10.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:10.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:10.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:10.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:11.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:11.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:11.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:11.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:11.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:28:11.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:11.859255+0000) 2019-10-18T06:28:12.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:12.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:12.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:12.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:13.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:13.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:13.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:13.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:13.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:13.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:14.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:14.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:14.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:14.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:15.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:15.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:15.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:15.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:16.063 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:28:16.058+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:16.061556+0000) 2019-10-18T06:28:16.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:16.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:16.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:16.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:17.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:17.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:17.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:18.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:18.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:18.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:18.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:18.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:18.724+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:19.054 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:28:19.050+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:19.053361+0000) 2019-10-18T06:28:19.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:19.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:19.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:19.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:19.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:28:19.820+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:19.822099+0000) 2019-10-18T06:28:20.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:20.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:20.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:20.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:21.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:21.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:21.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:21.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:21.860 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:28:21.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:21.859354+0000) 2019-10-18T06:28:22.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:22.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:22.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:22.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:23.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:23.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:23.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:23.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:23.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:23.724+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:24.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:24.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:24.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:24.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:25.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:25.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:25.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:25.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:26.063 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:28:26.057+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:26.061704+0000) 2019-10-18T06:28:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:26.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:26.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:26.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:27.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:27.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:27.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:27.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:28.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:28.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:28.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:28.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:28.729 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:28.724+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:29.054 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:28:29.049+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:29.053564+0000) 2019-10-18T06:28:29.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:29.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:29.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:29.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:29.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:28:29.820+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:29.822200+0000) 2019-10-18T06:28:30.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:30.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:30.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:30.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:31.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:31.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:31.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:31.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:31.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:28:31.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:31.859453+0000) 2019-10-18T06:28:32.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:32.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:32.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:32.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:33.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:33.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:33.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:33.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:33.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:33.724+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:34.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:34.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:34.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:34.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:35.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:35.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:35.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:35.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:36.063 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:28:36.057+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:36.061893+0000) 2019-10-18T06:28:36.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:36.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:36.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:36.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:37.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:37.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:37.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:37.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:38.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:38.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:38.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:38.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:38.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:38.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:39.055 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:28:39.049+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:39.053740+0000) 2019-10-18T06:28:39.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:39.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:39.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:39.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:39.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:28:39.820+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:39.822300+0000) 2019-10-18T06:28:40.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:40.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:40.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:40.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:41.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:41.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:41.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:41.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:41.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:28:41.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:41.859550+0000) 2019-10-18T06:28:42.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:42.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:42.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:42.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:43.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:43.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:43.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:43.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:43.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:43.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:44.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:44.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:44.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:44.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:45.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:45.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:45.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:45.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:46.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:28:46.057+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:46.062084+0000) 2019-10-18T06:28:46.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:46.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:46.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:46.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:47.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:47.600+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:47.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:47.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:48.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:48.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:48.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:48.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:48.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:48.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:49.055 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:28:49.049+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:49.053925+0000) 2019-10-18T06:28:49.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:49.592+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:49.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:49.592+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:49.823 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:28:49.820+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:49.822397+0000) 2019-10-18T06:28:50.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:50.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:50.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:50.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:51.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:51.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:51.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:51.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:51.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:28:51.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:51.859648+0000) 2019-10-18T06:28:52.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:52.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:52.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:52.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:53.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:53.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:53.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:53.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:53.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:53.727+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:54.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:54.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:54.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:54.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:55.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:55.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:55.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:55.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:56.063 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:28:56.061+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:56.062237+0000) 2019-10-18T06:28:56.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:56.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:56.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:56.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:57.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:57.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:57.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:57.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:58.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:58.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:58.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:58.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:58.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:28:58.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:28:59.055 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:28:59.049+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:59.054101+0000) 2019-10-18T06:28:59.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:59.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:28:59.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:28:59.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:28:59.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:28:59.819+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:28:59.822504+0000) 2019-10-18T06:29:00.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:00.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:00.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:00.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:01.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:01.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:01.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:01.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:01.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:29:01.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:01.859743+0000) 2019-10-18T06:29:02.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:02.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:02.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:02.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:03.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:03.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:03.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:03.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:03.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:03.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:04.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:04.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:04.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:04.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:05.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:05.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:05.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:05.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:06.064 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:29:06.061+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:06.062440+0000) 2019-10-18T06:29:06.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:06.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:06.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:06.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:07.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:07.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:07.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:07.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:08.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:08.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:08.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:08.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:08.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:08.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:09.055 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:29:09.053+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:09.054285+0000) 2019-10-18T06:29:09.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:09.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:09.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:09.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:09.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:09.819+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:09.822609+0000) 2019-10-18T06:29:10.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:10.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:10.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:10.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:11.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:11.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:11.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:11.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:11.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:29:11.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:11.859842+0000) 2019-10-18T06:29:12.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:12.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:12.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:12.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:13.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:13.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:13.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:13.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:13.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:13.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:14.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:14.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:14.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:14.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:15.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:15.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:15.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:15.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:16.064 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:29:16.061+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:16.062627+0000) 2019-10-18T06:29:16.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:16.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:16.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:16.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:17.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:17.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:17.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:17.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:18.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:18.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:18.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:18.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:18.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:18.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:19.056 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:29:19.053+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:19.054450+0000) 2019-10-18T06:29:19.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:19.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:19.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:19.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:19.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:19.819+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:19.822713+0000) 2019-10-18T06:29:20.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:20.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:20.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:20.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:21.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:21.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:21.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:21.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:21.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:29:21.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:21.859947+0000) 2019-10-18T06:29:22.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:22.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:22.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:22.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:23.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:23.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:23.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:23.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:23.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:24.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:24.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:24.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:24.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:25.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:25.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:25.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:25.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:26.066 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:29:26.061+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:26.062808+0000) 2019-10-18T06:29:26.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:26.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:26.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:26.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:27.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:27.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:27.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:27.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:28.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:28.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:28.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:28.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:28.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:28.734+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:29.055 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:29:29.053+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:29.054617+0000) 2019-10-18T06:29:29.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:29.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:29.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:29.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:29.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:29.818+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:29.822824+0000) 2019-10-18T06:29:30.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:30.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:30.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:30.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:31.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:31.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:31.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:31.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:31.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:29:31.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:31.860046+0000) 2019-10-18T06:29:32.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:32.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:32.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:32.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:33.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:33.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:33.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:33.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:33.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:33.734+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:34.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:34.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:34.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:34.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:35.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:35.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:35.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:35.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:36.064 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:29:36.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:36.062995+0000) 2019-10-18T06:29:36.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:36.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:36.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:36.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:37.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:37.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:37.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:37.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:38.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:38.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:38.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:38.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:38.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:38.734+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:39.056 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:29:39.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:39.054788+0000) 2019-10-18T06:29:39.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:39.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:39.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:39.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:39.824 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:39.818+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:39.822925+0000) 2019-10-18T06:29:39.825 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:39.818+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:39.823883+0000) 2019-10-18T06:29:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:40.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:40.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:40.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:41.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:41.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:41.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:41.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:41.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:29:41.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:41.860145+0000) 2019-10-18T06:29:42.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:42.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:42.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:42.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:43.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:43.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:43.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:43.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:43.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:43.734+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:44.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:44.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:44.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:44.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:44.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:44.318+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:44.323116+0000) 2019-10-18T06:29:45.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:45.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:45.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:45.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:46.064 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:29:46.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:46.063141+0000) 2019-10-18T06:29:46.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:46.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:46.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:46.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:47.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:47.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:47.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:47.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:48.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:48.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:48.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:48.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:48.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:48.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:49.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:29:49.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:49.054942+0000) 2019-10-18T06:29:49.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:49.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:49.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:49.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:50.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:50.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:50.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:50.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:51.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:51.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:51.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:51.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:51.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:29:51.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:51.860251+0000) 2019-10-18T06:29:52.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:52.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:52.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:52.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:53.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:53.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:53.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:53.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:53.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:53.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:54.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:54.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:54.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:54.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:54.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:29:54.318+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:54.323222+0000) 2019-10-18T06:29:55.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:55.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:55.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:55.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:56.064 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:29:56.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:56.063322+0000) 2019-10-18T06:29:56.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:56.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:56.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:56.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:57.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:57.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:57.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:57.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:58.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:58.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:58.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:58.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:29:58.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:29:58.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:29:59.056 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:29:59.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:29:59.055109+0000) 2019-10-18T06:29:59.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:59.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:29:59.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:29:59.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:00.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:00.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:00.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:00.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:01.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:01.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:01.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:01.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:01.861 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:30:01.854+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:01.860346+0000) 2019-10-18T06:30:02.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:02.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:02.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:02.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:03.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:03.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:03.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:03.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:03.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:03.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:04.324 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:30:04.317+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:04.323330+0000) 2019-10-18T06:30:04.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:04.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:04.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:04.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:05.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:05.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:05.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:05.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:06.064 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:30:06.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:06.063506+0000) 2019-10-18T06:30:06.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:06.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:06.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:06.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:07.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:07.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:07.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:07.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:08.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:08.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:08.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:08.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:08.737+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:09.056 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:30:09.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:09.055293+0000) 2019-10-18T06:30:09.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:09.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:09.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:09.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:10.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:10.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:10.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:10.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:11.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:11.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:11.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:11.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:30:11.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:11.860441+0000) 2019-10-18T06:30:12.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:12.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:12.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:12.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:13.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:13.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:13.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:13.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:13.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:13.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:14.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:30:14.317+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:14.323434+0000) 2019-10-18T06:30:14.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:14.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:14.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:14.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:15.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:15.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:15.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:15.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:16.065 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:30:16.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:16.063685+0000) 2019-10-18T06:30:16.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:16.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:16.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:16.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:17.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:17.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:17.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:17.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:18.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:18.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:18.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:18.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:18.748 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:18.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:19.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:30:19.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:19.055469+0000) 2019-10-18T06:30:19.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:19.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:19.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:19.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:20.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:20.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:20.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:20.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:21.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:21.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:21.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:21.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:21.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:30:21.853+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:21.860541+0000) 2019-10-18T06:30:22.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:22.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:22.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:22.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:23.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:23.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:23.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:23.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:23.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:23.745+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:24.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:30:24.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:24.323541+0000) 2019-10-18T06:30:24.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:24.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:24.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:24.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:25.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:25.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:25.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:25.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:26.065 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:30:26.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:26.063869+0000) 2019-10-18T06:30:26.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:26.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:26.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:26.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:27.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:27.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:27.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:27.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:28.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:28.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:28.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:28.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:28.750 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:28.745+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:29.056 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:30:29.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:29.055653+0000) 2019-10-18T06:30:29.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:29.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:29.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:29.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:30.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:30.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:30.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:30.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:31.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:31.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:31.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:31.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:31.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:30:31.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:31.860638+0000) 2019-10-18T06:30:32.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:32.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:32.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:32.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:33.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:33.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:33.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:33.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:33.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:33.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:34.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:30:34.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:34.323650+0000) 2019-10-18T06:30:34.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:34.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:34.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:34.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:35.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:35.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:35.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:35.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:36.065 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:30:36.060+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:36.064061+0000) 2019-10-18T06:30:36.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:36.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:36.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:36.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:37.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:37.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:37.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:37.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:38.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:38.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:38.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:38.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:38.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:38.748+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:39.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:30:39.052+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:39.055817+0000) 2019-10-18T06:30:39.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:39.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:39.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:39.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:40.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:40.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:40.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:40.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:41.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:41.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:41.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:41.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:41.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:30:41.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:41.860739+0000) 2019-10-18T06:30:42.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:42.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:42.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:42.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:43.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:43.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:43.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:43.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:43.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:43.748+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:44.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:44.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:44.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:44.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:44.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:30:44.320+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:44.323758+0000) 2019-10-18T06:30:45.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:45.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:45.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:45.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:46.065 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:30:46.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:46.064253+0000) 2019-10-18T06:30:46.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:46.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:46.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:46.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:47.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:47.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:47.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:47.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:48.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:48.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:48.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:48.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:48.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:48.748+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:49.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:30:49.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:49.055981+0000) 2019-10-18T06:30:49.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:49.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:49.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:49.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:50.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:50.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:50.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:50.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:51.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:51.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:51.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:51.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:51.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:30:51.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:51.860842+0000) 2019-10-18T06:30:52.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:52.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:52.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:52.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:53.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:53.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:53.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:53.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:53.748+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:54.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:54.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:54.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:54.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:54.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:30:54.320+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:54.323863+0000) 2019-10-18T06:30:55.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:55.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:55.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:55.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:56.065 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:30:56.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:56.064456+0000) 2019-10-18T06:30:56.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:56.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:56.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:56.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:57.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:57.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:57.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:57.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:58.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:58.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:58.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:58.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:30:58.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:30:58.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:30:59.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:30:59.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:30:59.056170+0000) 2019-10-18T06:30:59.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:59.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:30:59.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:30:59.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:00.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:00.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:00.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:00.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:01.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:01.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:01.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:01.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:01.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:31:01.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:01.860940+0000) 2019-10-18T06:31:02.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:02.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:02.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:02.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:03.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:03.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:03.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:03.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:03.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:03.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:04.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:04.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:04.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:04.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:04.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:31:04.320+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:04.323974+0000) 2019-10-18T06:31:05.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:05.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:05.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:05.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:06.066 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:31:06.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:06.064651+0000) 2019-10-18T06:31:06.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:06.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:06.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:06.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:07.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:07.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:07.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:07.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:08.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:08.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:08.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:08.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:08.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:08.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:09.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:31:09.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:09.056355+0000) 2019-10-18T06:31:09.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:09.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:09.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:09.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:10.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:10.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:10.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:10.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:11.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:11.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:11.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:11.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:11.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:31:11.856+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:11.861041+0000) 2019-10-18T06:31:12.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:12.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:12.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:12.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:13.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:13.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:13.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:13.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:13.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:13.751+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:14.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:14.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:14.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:14.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:14.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:31:14.319+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:14.324083+0000) 2019-10-18T06:31:15.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:15.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:15.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:15.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:16.066 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:31:16.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:16.064838+0000) 2019-10-18T06:31:16.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:16.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:16.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:16.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:17.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:17.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:17.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:17.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:18.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:18.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:18.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:18.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:18.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:18.751+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:19.057 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:31:19.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:19.056543+0000) 2019-10-18T06:31:19.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:19.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:19.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:19.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:20.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:20.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:20.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:20.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:21.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:21.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:21.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:21.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:21.862 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:31:21.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:21.861142+0000) 2019-10-18T06:31:22.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:22.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:22.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:22.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:23.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:23.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:23.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:23.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:23.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:23.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:24.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:24.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:24.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:24.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:24.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:31:24.319+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:24.324193+0000) 2019-10-18T06:31:25.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:25.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:25.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:25.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:26.066 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:31:26.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:26.065021+0000) 2019-10-18T06:31:26.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:26.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:26.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:26.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:27.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:27.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:27.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:27.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:28.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:28.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:28.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:28.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:28.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:28.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:29.058 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:31:29.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:29.056742+0000) 2019-10-18T06:31:29.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:29.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:29.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:29.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:30.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:30.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:30.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:30.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:31.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:31.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:31.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:31.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:31.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:31:31.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:31.861243+0000) 2019-10-18T06:31:32.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:32.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:32.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:32.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:33.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:33.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:33.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:33.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:33.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:33.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:34.325 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:31:34.319+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:34.324301+0000) 2019-10-18T06:31:34.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:34.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:34.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:34.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:35.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:35.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:35.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:35.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:36.066 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:31:36.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:36.065234+0000) 2019-10-18T06:31:36.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:36.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:36.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:36.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:37.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:37.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:37.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:37.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:38.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:38.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:38.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:38.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:39.058 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:31:39.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:39.056945+0000) 2019-10-18T06:31:39.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:39.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:39.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:39.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:40.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:40.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:40.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:40.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:41.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:41.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:41.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:41.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:41.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:31:41.855+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:41.861339+0000) 2019-10-18T06:31:42.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:42.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:42.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:42.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:43.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:43.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:43.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:43.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:43.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:43.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:44.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:44.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:44.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:44.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:44.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:31:44.319+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:44.324402+0000) 2019-10-18T06:31:45.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:45.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:45.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:45.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:46.066 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:31:46.059+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:46.065448+0000) 2019-10-18T06:31:46.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:46.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:46.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:46.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:47.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:47.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:47.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:47.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:48.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:48.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:48.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:48.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:48.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:49.058 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:31:49.051+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:49.057124+0000) 2019-10-18T06:31:49.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:49.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:49.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:49.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:50.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:50.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:50.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:50.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:51.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:51.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:51.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:51.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:51.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:31:51.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:51.861456+0000) 2019-10-18T06:31:52.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:52.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:52.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:52.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:53.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:53.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:53.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:53.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:53.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:53.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:54.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:54.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:54.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:54.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:54.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:31:54.318+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:54.324509+0000) 2019-10-18T06:31:55.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:55.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:55.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:55.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:56.067 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:31:56.058+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:56.065631+0000) 2019-10-18T06:31:56.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:56.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:56.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:56.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:57.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:57.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:57.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:57.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:58.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:58.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:58.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:58.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:31:58.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:31:58.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:31:59.058 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:31:59.050+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:31:59.057298+0000) 2019-10-18T06:31:59.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:59.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:31:59.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:31:59.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:00.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:00.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:00.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:00.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:01.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:01.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:01.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:01.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:01.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:32:01.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:01.861554+0000) 2019-10-18T06:32:02.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:02.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:02.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:02.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:03.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:03.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:03.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:03.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:03.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:03.758+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:04.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:04.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:04.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:04.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:04.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:32:04.318+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:04.324618+0000) 2019-10-18T06:32:05.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:05.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:05.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:05.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:06.067 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:32:06.058+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:06.065825+0000) 2019-10-18T06:32:06.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:06.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:06.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:06.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:07.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:07.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:07.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:07.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:08.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:08.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:08.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:08.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:08.767 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:08.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:09.059 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:32:09.050+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:09.057486+0000) 2019-10-18T06:32:09.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:09.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:09.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:09.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:10.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:10.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:10.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:10.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:11.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:11.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:11.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:11.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:11.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:32:11.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:11.861655+0000) 2019-10-18T06:32:12.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:12.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:12.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:12.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:13.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:13.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:13.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:13.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:13.768 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:13.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:14.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:14.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:14.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:14.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:14.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:32:14.318+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:14.324725+0000) 2019-10-18T06:32:15.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:15.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:15.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:15.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:16.067 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:32:16.058+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:16.066018+0000) 2019-10-18T06:32:16.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:16.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:16.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:16.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:17.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:17.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:17.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:17.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:18.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:18.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:18.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:18.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:18.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:18.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:19.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:19.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:19.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:19.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:19.059 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:32:19.050+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:19.057670+0000) 2019-10-18T06:32:20.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:19.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:20.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:19.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:20.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:20.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:20.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:20.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:21.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:32:21.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:21.861755+0000) 2019-10-18T06:32:22.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:22.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:22.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:22.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:23.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:23.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:23.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:23.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:23.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:23.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:24.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:24.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:24.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:24.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:24.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:32:24.317+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:24.324848+0000) 2019-10-18T06:32:25.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:25.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:25.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:25.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:26.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:26.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:26.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:26.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:26.067 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:32:26.062+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:26.066228+0000) 2019-10-18T06:32:26.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:26.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:26.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:26.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:28.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:28.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:28.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:28.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:28.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:28.761+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:29.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:29.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:29.059 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:32:29.050+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:29.057857+0000) 2019-10-18T06:32:30.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:30.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:30.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:30.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:31.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:31.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:31.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:31.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:31.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:32:31.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:31.861850+0000) 2019-10-18T06:32:32.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:32.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:21:57.123128+0000 (oldest deadline 2019-10-18T06:22:17.123128+0000) 2019-10-18T06:32:32.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:32.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:33.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:33.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:33.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:33.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:34.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:34.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:34.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:32:34.317+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:34.324957+0000) 2019-10-18T06:32:35.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:35.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:36.068 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:32:36.062+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:36.066438+0000) 2019-10-18T06:32:36.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:36.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:37.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:37.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:38.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:38.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:38.772 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:38.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:39.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:39.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:39.062 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:32:39.050+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:39.058041+0000) 2019-10-18T06:32:40.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:40.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:41.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:41.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:41.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:32:41.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:41.861950+0000) 2019-10-18T06:32:41.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:41.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:42.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:42.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:43.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:43.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:43.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:43.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:44.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:32:44.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:44.325070+0000) 2019-10-18T06:32:45.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:45.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:45.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:45.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:46.068 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:32:46.062+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:46.066647+0000) 2019-10-18T06:32:46.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:46.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:48.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:48.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:48.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:48.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:49.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:49.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:49.059 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:32:49.054+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:49.058224+0000) 2019-10-18T06:32:50.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:50.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:51.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:51.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:51.863 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:32:51.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:51.862048+0000) 2019-10-18T06:32:52.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:52.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:53.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:53.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:53.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:53.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:54.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:54.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:54.326 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:32:54.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:54.325188+0000) 2019-10-18T06:32:55.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:55.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:32:55.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:55.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:56.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:56.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:32:56.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:56.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:56.068 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:32:56.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:56.066833+0000) 2019-10-18T06:32:56.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:56.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:32:57.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:56.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:58.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:58.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:32:58.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:58.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:58.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:32:58.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:32:59.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:59.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:32:59.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:32:59.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:32:59.059 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:32:59.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:32:59.058411+0000) 2019-10-18T06:33:00.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:00.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:00.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:00.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:01.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:01.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:01.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:01.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:01.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:33:01.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:01.862145+0000) 2019-10-18T06:33:02.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:02.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:02.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:02.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:03.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:03.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:03.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:03.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:03.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:03.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:04.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:04.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:04.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:04.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:04.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:33:04.320+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:04.325301+0000) 2019-10-18T06:33:05.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:05.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:05.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:05.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:06.068 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:33:06.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:06.067018+0000) 2019-10-18T06:33:06.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:06.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:06.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:06.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:07.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:07.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:07.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:07.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:08.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:08.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:08.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:08.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:08.777 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:08.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:09.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:09.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:09.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:09.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:09.059 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:33:09.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:09.058604+0000) 2019-10-18T06:33:09.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:09.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:09.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:09.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:10.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:10.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:10.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:10.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:11.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:33:11.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:11.862243+0000) 2019-10-18T06:33:11.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:11.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:11.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:11.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:12.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:12.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:12.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:12.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:13.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:13.772+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:13.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:13.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:13.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:13.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:14.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:33:14.320+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:14.325411+0000) 2019-10-18T06:33:14.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:14.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:14.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:14.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:15.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:15.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:15.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:15.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:16.068 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:33:16.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:16.067200+0000) 2019-10-18T06:33:16.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:16.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:16.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:16.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:17.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:17.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:17.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:17.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:18.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:18.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:18.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:18.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:18.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:18.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:19.060 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:33:19.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:19.058782+0000) 2019-10-18T06:33:19.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:19.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:19.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:19.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:21.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:21.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:21.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:21.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:21.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:33:21.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:21.862348+0000) 2019-10-18T06:33:22.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:22.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:22.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:22.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:23.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:23.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:23.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:23.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:23.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:23.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:24.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:24.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:24.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:24.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:24.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:33:24.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:24.325530+0000) 2019-10-18T06:33:25.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:25.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:25.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:25.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:26.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:26.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:26.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:26.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:26.068 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:33:26.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:26.067382+0000) 2019-10-18T06:33:27.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:27.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:27.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:27.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:28.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:28.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:28.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:28.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:28.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:28.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:29.060 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:33:29.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:29.058972+0000) 2019-10-18T06:33:29.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:29.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:29.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:29.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:30.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:30.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:30.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:30.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:31.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:31.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:31.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:31.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:31.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:33:31.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:31.862447+0000) 2019-10-18T06:33:32.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:32.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:32.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:32.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:33.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:33.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:33.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:33.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:33.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:33.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:34.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:34.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:34.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:34.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:34.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:33:34.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:34.325638+0000) 2019-10-18T06:33:35.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:35.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:35.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:35.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:36.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:33:36.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:36.067563+0000) 2019-10-18T06:33:36.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:36.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:36.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:36.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:37.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:37.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:37.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:37.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:38.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:38.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:38.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:38.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:38.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:38.775+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:39.060 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:33:39.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:39.059149+0000) 2019-10-18T06:33:39.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:39.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:39.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:39.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:40.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:40.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:40.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:40.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:41.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:41.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:41.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:41.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:41.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:33:41.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:41.862546+0000) 2019-10-18T06:33:42.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:42.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:42.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:42.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:43.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:43.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:43.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:43.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:43.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:43.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:44.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:33:44.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:44.325748+0000) 2019-10-18T06:33:44.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:44.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:44.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:44.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:45.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:45.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:45.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:45.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:46.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:33:46.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:46.067745+0000) 2019-10-18T06:33:46.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:46.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:46.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:46.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:47.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:47.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:47.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:47.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:48.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:48.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:48.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:48.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:48.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:48.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:49.061 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:33:49.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:49.059339+0000) 2019-10-18T06:33:49.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:49.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:49.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:49.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:50.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:50.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:50.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:50.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:51.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:51.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:51.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:51.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:51.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:33:51.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:51.862651+0000) 2019-10-18T06:33:52.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:52.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:52.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:52.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:53.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:53.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:53.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:53.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:53.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:53.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:54.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:54.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:54.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:54.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:54.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:33:54.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:54.325859+0000) 2019-10-18T06:33:55.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:55.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:55.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:55.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:56.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:33:56.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:56.067934+0000) 2019-10-18T06:33:56.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:56.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:56.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:56.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:57.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:57.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:57.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:57.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:58.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:58.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:58.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:58.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:33:58.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:33:58.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:33:59.061 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:33:59.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:33:59.059508+0000) 2019-10-18T06:33:59.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:59.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:33:59.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:33:59.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:00.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:00.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:00.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:00.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:01.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:01.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:01.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:01.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:01.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:34:01.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:01.862751+0000) 2019-10-18T06:34:02.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:02.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:02.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:02.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:03.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:03.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:03.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:03.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:03.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:03.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:04.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:34:04.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:04.325973+0000) 2019-10-18T06:34:04.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:04.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:04.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:04.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:05.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:05.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:05.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:05.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:06.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:34:06.065+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:06.068121+0000) 2019-10-18T06:34:06.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:06.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:06.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:06.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:07.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:07.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:07.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:07.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:08.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:08.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:08.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:08.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:08.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:08.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:09.061 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:34:09.057+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:09.059706+0000) 2019-10-18T06:34:09.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:09.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:09.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:09.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:10.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:10.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:10.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:10.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:11.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:11.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:11.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:11.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:34:11.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:11.862850+0000) 2019-10-18T06:34:12.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:12.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:12.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:12.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:13.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:13.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:13.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:13.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:13.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:13.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:14.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:34:14.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:14.326080+0000) 2019-10-18T06:34:14.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:14.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:14.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:14.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:15.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:15.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:15.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:15.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:16.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:34:16.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:16.068315+0000) 2019-10-18T06:34:16.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:16.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:16.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:16.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:17.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:17.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:17.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:17.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:18.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:18.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:18.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:18.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:18.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:18.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:19.061 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:34:19.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:19.059888+0000) 2019-10-18T06:34:19.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:19.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:19.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:19.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:20.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:20.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:20.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:20.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:21.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:21.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:21.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:21.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:21.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:34:21.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:21.862947+0000) 2019-10-18T06:34:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:22.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:22.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:22.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:23.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:23.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:23.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:23.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:23.790 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:23.782+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:24.327 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:34:24.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:24.326187+0000) 2019-10-18T06:34:24.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:24.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:24.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:24.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:25.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:25.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:25.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:25.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:26.069 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:34:26.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:26.068501+0000) 2019-10-18T06:34:26.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:26.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:26.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:26.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:27.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:27.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:27.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:27.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:28.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:28.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:28.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:28.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:28.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:28.786+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:29.061 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:34:29.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:29.060086+0000) 2019-10-18T06:34:29.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:29.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:29.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:29.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:30.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:30.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:30.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:30.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:31.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:31.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:31.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:31.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:31.864 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:34:31.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:31.863047+0000) 2019-10-18T06:34:32.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:32.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:32.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:32.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:33.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:33.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:33.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:33.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:33.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:33.786+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:34.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:34:34.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:34.326306+0000) 2019-10-18T06:34:34.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:34.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:34.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:34.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:35.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:35.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:35.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:35.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:36.070 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:34:36.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:36.068691+0000) 2019-10-18T06:34:36.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:36.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:36.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:36.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:37.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:37.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:37.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:37.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:38.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:38.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:38.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:38.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:38.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:38.786+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:39.061 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:34:39.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:39.060309+0000) 2019-10-18T06:34:39.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:39.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:39.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:39.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:40.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:40.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:40.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:40.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:41.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:41.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:41.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:41.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:41.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:34:41.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:41.863147+0000) 2019-10-18T06:34:42.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:42.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:42.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:42.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:43.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:43.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:43.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:43.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:43.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:43.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:44.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:34:44.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:44.326415+0000) 2019-10-18T06:34:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:44.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:44.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:44.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:45.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:45.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:45.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:46.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:34:46.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:46.068874+0000) 2019-10-18T06:34:46.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:46.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:46.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:46.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:47.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:47.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:47.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:47.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:48.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:48.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:48.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:48.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:48.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:48.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:49.062 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:34:49.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:49.060503+0000) 2019-10-18T06:34:49.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:49.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:49.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:49.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:50.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:50.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:50.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:50.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:51.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:51.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:51.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:51.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:51.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:34:51.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:51.863249+0000) 2019-10-18T06:34:52.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:52.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:52.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:52.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:53.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:53.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:53.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:53.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:53.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:53.785+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:54.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:34:54.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:54.326525+0000) 2019-10-18T06:34:54.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:54.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:54.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:54.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:55.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:55.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:55.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:55.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:56.070 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:34:56.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:56.069062+0000) 2019-10-18T06:34:56.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:56.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:56.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:56.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:57.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:57.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:57.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:57.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:58.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:58.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:58.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:58.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:34:58.796 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:34:58.789+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:34:59.062 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:34:59.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:34:59.060689+0000) 2019-10-18T06:34:59.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:59.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:34:59.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:34:59.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:00.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:00.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:00.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:00.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:01.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:01.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:01.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:01.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:01.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:35:01.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:01.863346+0000) 2019-10-18T06:35:02.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:02.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:02.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:02.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:03.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:03.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:03.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:03.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:03.797 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:03.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:04.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:35:04.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:04.326643+0000) 2019-10-18T06:35:04.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:04.653+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:04.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:04.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:05.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:05.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:05.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:05.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:06.070 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:35:06.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:06.069274+0000) 2019-10-18T06:35:06.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:06.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:06.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:06.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:07.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:07.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:07.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:07.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:08.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:08.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:08.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:08.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:08.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:08.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:09.062 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:35:09.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:09.060879+0000) 2019-10-18T06:35:09.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:09.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:09.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:09.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:10.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:10.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:10.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:10.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:11.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:11.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:11.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:11.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:11.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:35:11.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:11.863441+0000) 2019-10-18T06:35:12.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:12.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:12.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:12.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:13.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:13.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:13.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:13.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:13.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:13.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:14.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:35:14.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:14.326758+0000) 2019-10-18T06:35:14.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:14.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:14.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:14.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:15.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:15.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:15.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:15.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:16.072 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:35:16.064+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:16.069488+0000) 2019-10-18T06:35:16.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:16.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:16.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:16.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:17.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:17.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:17.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:17.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:18.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:18.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:18.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:18.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:18.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:18.796+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:19.062 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:35:19.056+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:19.061071+0000) 2019-10-18T06:35:19.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:19.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:19.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:19.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:20.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:20.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:20.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:20.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:21.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:21.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:21.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:21.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:21.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:35:21.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:21.863542+0000) 2019-10-18T06:35:22.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:22.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:22.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:22.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:23.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:23.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:23.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:23.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:23.800 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:23.796+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:24.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:35:24.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:24.326867+0000) 2019-10-18T06:35:24.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:24.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:24.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:24.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:25.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:25.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:25.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:25.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:26.071 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:35:26.063+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:26.069675+0000) 2019-10-18T06:35:26.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:26.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:26.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:26.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:27.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:27.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:27.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:27.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:28.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:28.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:28.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:28.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:28.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:28.796+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:29.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:35:29.055+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:29.061264+0000) 2019-10-18T06:35:29.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:29.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:29.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:29.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:30.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:30.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:30.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:30.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:31.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:31.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:31.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:31.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:31.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:35:31.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:31.863641+0000) 2019-10-18T06:35:32.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:32.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:32.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:32.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:33.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:33.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:33.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:33.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:33.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:33.796+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:34.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:35:34.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:34.326971+0000) 2019-10-18T06:35:34.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:34.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:34.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:34.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:35.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:35.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:35.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:35.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:36.079 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:35:36.063+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:36.069857+0000) 2019-10-18T06:35:36.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:36.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:36.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:36.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:37.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:37.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:37.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:37.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:38.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:38.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:38.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:38.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:38.803 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:38.800+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:39.062 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:35:39.055+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:39.061449+0000) 2019-10-18T06:35:39.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:39.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:39.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:39.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:40.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:40.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:40.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:40.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:41.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:41.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:41.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:41.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:41.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:35:41.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:41.863738+0000) 2019-10-18T06:35:42.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:42.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:42.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:42.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:43.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:43.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:43.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:43.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:43.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:43.800+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:44.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:35:44.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:44.327076+0000) 2019-10-18T06:35:44.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:44.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:44.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:44.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:45.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:45.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:45.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:45.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:46.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:35:46.063+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:46.070039+0000) 2019-10-18T06:35:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:46.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:46.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:46.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:47.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:47.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:47.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:47.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:48.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:48.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:48.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:48.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:48.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:48.800+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:49.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:35:49.055+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:49.061647+0000) 2019-10-18T06:35:49.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:49.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:49.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:49.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:50.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:50.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:50.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:50.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:51.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:51.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:51.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:51.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:51.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:35:51.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:51.863838+0000) 2019-10-18T06:35:52.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:52.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:52.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:52.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:53.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:53.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:53.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:53.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:53.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:53.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:54.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:35:54.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:54.327183+0000) 2019-10-18T06:35:54.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:54.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:54.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:54.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:55.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:55.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:55.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:55.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:56.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:35:56.067+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:56.070218+0000) 2019-10-18T06:35:56.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:56.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:56.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:56.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:57.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:57.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:57.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:57.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:58.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:58.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:58.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:35:58.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:35:58.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:35:59.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:35:59.055+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:35:59.061824+0000) 2019-10-18T06:35:59.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:59.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:35:59.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:35:59.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:00.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:00.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:00.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:00.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:01.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:01.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:01.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:01.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:01.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:36:01.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:01.863933+0000) 2019-10-18T06:36:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:02.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:02.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:02.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:03.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:03.799+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:03.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:03.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:03.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:03.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:04.328 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:36:04.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:04.327279+0000) 2019-10-18T06:36:04.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:04.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:04.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:04.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:05.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:05.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:05.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:05.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:06.072 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:36:06.067+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:06.070405+0000) 2019-10-18T06:36:06.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:06.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:06.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:06.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:07.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:07.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:07.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:07.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:08.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:08.803+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:08.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:08.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:08.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:08.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:09.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:36:09.055+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:09.061991+0000) 2019-10-18T06:36:09.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:09.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:09.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:09.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:10.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:10.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:10.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:10.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 516 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:11.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:11.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:11.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:11.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:11.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:36:11.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:11.864035+0000) 2019-10-18T06:36:12.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:12.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:12.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:12.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:13.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:13.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:13.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:13.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:13.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:13.803+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:14.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:36:14.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:14.327386+0000) 2019-10-18T06:36:14.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:14.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:14.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:14.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:15.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:15.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:15.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:15.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:16.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:36:16.067+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:16.070590+0000) 2019-10-18T06:36:16.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:16.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:16.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:16.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:17.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:17.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:17.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:17.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:18.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:18.803+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:18.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:18.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:18.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:18.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:19.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:36:19.059+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:19.062198+0000) 2019-10-18T06:36:19.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:19.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:19.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:19.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:20.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:20.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:20.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:20.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:21.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:21.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:21.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:21.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:21.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:36:21.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:21.864132+0000) 2019-10-18T06:36:22.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:22.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:22.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:22.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:23.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:23.803+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:23.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:23.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:23.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:23.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:24.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:36:24.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:24.327488+0000) 2019-10-18T06:36:24.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:24.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:24.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:24.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:25.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:25.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:25.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:25.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:26.072 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:36:26.067+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:26.070778+0000) 2019-10-18T06:36:26.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:26.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:26.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:26.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:27.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:27.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:27.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:27.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:28.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:28.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:28.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:28.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:28.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:28.806+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:29.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:36:29.059+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:29.062390+0000) 2019-10-18T06:36:29.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:29.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:29.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:29.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:30.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:30.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:30.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:30.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:31.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:31.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:31.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:31.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:31.865 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:36:31.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:31.864244+0000) 2019-10-18T06:36:32.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:32.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:32.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:32.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:33.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:33.806+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:33.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:33.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:33.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:33.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:34.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:36:34.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:34.327594+0000) 2019-10-18T06:36:34.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:34.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:34.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:35.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:35.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:35.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:35.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:36.072 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:36:36.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:36.070967+0000) 2019-10-18T06:36:36.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:36.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:36.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:36.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:37.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:37.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:37.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:37.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:38.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:38.806+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:38.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:38.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:38.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:38.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:39.063 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:36:39.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:39.062558+0000) 2019-10-18T06:36:39.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:39.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:39.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:39.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:40.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:40.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:40.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:40.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:41.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:36:41.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:41.864341+0000) 2019-10-18T06:36:41.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:41.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:41.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:41.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:42.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:42.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:42.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:42.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:43.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:43.806+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:43.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:43.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:43.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:43.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:44.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:36:44.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:44.327700+0000) 2019-10-18T06:36:44.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:44.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:44.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:44.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:45.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:45.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:45.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:45.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:46.072 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:36:46.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:46.071154+0000) 2019-10-18T06:36:46.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:46.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:46.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:46.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:47.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:47.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:47.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:47.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:48.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:48.806+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:48.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:48.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:48.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:48.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:49.064 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:36:49.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:49.062740+0000) 2019-10-18T06:36:49.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:49.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:49.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:49.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:50.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:50.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:50.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:50.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:51.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:36:51.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:51.864437+0000) 2019-10-18T06:36:51.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:51.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:51.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:51.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:52.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:52.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:52.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:52.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:53.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:53.810+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:53.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:53.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:53.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:53.950+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:54.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:36:54.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:54.327803+0000) 2019-10-18T06:36:55.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:54.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:55.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:54.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:55.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:55.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:55.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:55.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:56.072 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:36:56.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:56.071295+0000) 2019-10-18T06:36:57.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:57.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:57.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:57.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:58.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:58.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:58.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:58.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:58.816 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:36:58.810+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:36:59.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:59.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:36:59.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:36:59.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:36:59.064 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:36:59.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:36:59.062924+0000) 2019-10-18T06:37:00.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:00.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:00.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:00.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:01.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:01.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:01.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:01.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:01.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:37:01.858+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:01.864537+0000) 2019-10-18T06:37:02.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:02.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:02.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:02.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:03.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:03.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:03.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:03.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:03.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:03.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:04.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:04.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:04.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:04.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:04.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:37:04.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:04.327919+0000) 2019-10-18T06:37:05.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:05.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:05.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:05.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:05.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:05.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:06.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:37:06.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:06.071466+0000) 2019-10-18T06:37:06.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:06.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:06.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:06.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:07.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:07.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:07.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:07.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:08.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:08.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:08.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:08.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:08.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:08.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:09.064 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:37:09.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:09.063055+0000) 2019-10-18T06:37:09.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:09.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:09.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:09.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:10.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:10.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:10.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:10.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:11.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:37:11.857+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:11.864633+0000) 2019-10-18T06:37:12.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:12.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:12.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:12.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:13.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:13.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:13.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:13.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:13.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:13.809+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:14.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:14.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:14.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:14.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:14.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:37:14.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:14.328023+0000) 2019-10-18T06:37:14.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:14.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:14.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:15.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:15.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:15.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:15.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:16.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:37:16.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:16.071651+0000) 2019-10-18T06:37:16.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:16.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:16.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:16.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:17.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:17.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:17.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:17.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:18.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:18.813+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:18.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:18.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:18.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:18.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:19.064 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:37:19.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:19.063218+0000) 2019-10-18T06:37:19.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:19.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:19.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:19.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:20.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:20.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:20.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:20.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:21.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:37:21.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:21.864734+0000) 2019-10-18T06:37:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:22.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:22.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:22.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:23.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:23.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:23.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:23.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:23.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:23.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:24.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:24.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:24.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:24.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:37:24.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:24.328126+0000) 2019-10-18T06:37:25.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:25.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:25.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:25.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:26.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:37:26.066+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:26.071793+0000) 2019-10-18T06:37:26.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:26.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:26.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:26.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:27.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:27.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:27.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:27.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:28.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:28.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:28.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:28.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:28.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:28.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:29.064 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:37:29.058+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:29.063399+0000) 2019-10-18T06:37:29.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:29.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:29.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:29.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:30.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:30.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:30.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:30.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:31.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:31.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:31.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:31.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:31.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:37:31.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:31.864834+0000) 2019-10-18T06:37:32.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:32.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:32.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:32.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:32.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:32.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:32.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:32.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:33.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:33.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:34.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:34.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:34.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:34.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:34.329 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:37:34.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:34.328236+0000) 2019-10-18T06:37:34.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:34.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:34.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:34.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:36.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:36.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:36.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:36.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:36.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:37:36.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:36.071976+0000) 2019-10-18T06:37:37.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:37.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:37.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:37.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:38.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:38.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:38.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:38.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:38.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:38.820+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:39.065 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:37:39.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:39.063576+0000) 2019-10-18T06:37:39.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:39.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:39.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:39.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:40.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:40.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:40.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:40.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:41.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:41.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:41.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:41.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:41.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:37:41.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:41.864933+0000) 2019-10-18T06:37:42.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:42.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:42.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:42.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:43.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:43.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:43.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:43.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:43.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:43.820+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:44.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:44.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:44.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:44.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:44.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:37:44.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:44.328343+0000) 2019-10-18T06:37:45.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:45.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:45.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:45.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:46.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:46.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:46.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:46.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:46.081 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:37:46.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:46.072169+0000) 2019-10-18T06:37:47.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:47.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:47.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:47.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:48.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:48.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:48.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:48.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:48.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:48.820+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:49.065 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:37:49.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:49.063754+0000) 2019-10-18T06:37:49.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:49.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:49.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:49.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:50.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:50.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:50.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:50.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:51.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:51.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:51.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:51.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:51.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:37:51.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:51.865033+0000) 2019-10-18T06:37:52.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:52.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:52.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:52.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:53.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:52.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:53.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:52.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:53.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:53.820+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:54.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:54.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:54.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:54.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:54.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:37:54.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:54.328446+0000) 2019-10-18T06:37:55.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:55.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:55.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:55.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:56.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:56.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:56.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:56.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:56.073 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:37:56.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:56.072362+0000) 2019-10-18T06:37:57.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:57.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:57.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:57.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:58.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:58.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:58.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:58.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:37:58.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:37:58.820+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:37:59.068 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:37:59.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:37:59.063964+0000) 2019-10-18T06:37:59.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:59.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:37:59.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:37:59.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:00.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:00.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:00.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:00.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:01.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:01.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:01.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:01.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:01.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:38:01.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:01.865135+0000) 2019-10-18T06:38:02.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:02.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:02.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:02.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:03.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:03.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:03.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:03.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:03.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:03.824+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:04.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:04.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:04.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:04.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:04.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:38:04.324+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:04.328554+0000) 2019-10-18T06:38:05.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:05.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:05.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:05.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:06.074 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:38:06.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:06.072550+0000) 2019-10-18T06:38:06.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:06.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:06.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:06.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:07.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:07.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:07.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:07.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:08.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:08.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:08.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:08.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:08.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:08.824+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:09.065 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:38:09.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:09.064182+0000) 2019-10-18T06:38:09.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:09.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:09.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:09.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:10.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:10.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:10.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:10.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:11.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:11.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:11.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:11.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:11.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:38:11.860+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:11.865232+0000) 2019-10-18T06:38:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:12.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:12.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:12.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:13.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:13.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:13.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:13.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:13.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:13.823+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:14.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:14.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:14.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:14.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:14.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:38:14.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:14.328661+0000) 2019-10-18T06:38:15.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:15.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:15.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:15.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:16.074 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:38:16.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:16.072730+0000) 2019-10-18T06:38:16.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:16.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:16.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:16.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:17.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:17.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:17.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:17.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:18.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:18.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:18.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:18.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:18.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:18.823+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:19.065 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:38:19.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:19.064369+0000) 2019-10-18T06:38:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:19.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:19.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:20.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:20.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:20.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:20.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:21.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:21.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:21.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:21.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:21.866 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:38:21.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:21.865329+0000) 2019-10-18T06:38:22.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:22.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:22.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:22.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:23.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:23.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:23.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:23.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:23.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:23.823+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:24.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:24.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:24.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:24.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:24.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:38:24.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:24.328769+0000) 2019-10-18T06:38:25.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:25.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:25.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:25.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:26.074 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:38:26.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:26.072912+0000) 2019-10-18T06:38:26.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:26.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:26.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:26.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:27.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:27.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:27.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:27.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:28.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:28.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:28.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:28.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:28.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:28.827+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:29.066 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:38:29.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:29.064565+0000) 2019-10-18T06:38:29.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:29.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:29.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:29.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:30.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:30.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:30.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:30.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:31.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:31.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:31.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:31.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:31.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:38:31.859+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:31.865428+0000) 2019-10-18T06:38:32.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:32.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:32.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:32.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:33.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:33.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:33.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:33.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:33.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:33.827+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:34.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:34.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:34.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:34.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:34.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:38:34.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:34.328882+0000) 2019-10-18T06:38:35.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:35.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:35.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:35.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:36.074 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:38:36.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:36.073092+0000) 2019-10-18T06:38:36.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:36.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:36.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:36.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:37.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:37.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:37.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:37.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:38.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:38.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:38.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:38.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:38.833 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:38.827+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:39.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:38:39.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:39.064755+0000) 2019-10-18T06:38:39.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:39.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:39.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:39.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:40.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:40.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:40.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:40.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:41.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:41.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:41.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:41.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:41.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:38:41.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:41.865606+0000) 2019-10-18T06:38:42.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:42.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:42.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:42.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:43.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:43.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:43.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:43.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:43.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:43.827+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:44.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:38:44.323+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:44.328998+0000) 2019-10-18T06:38:44.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:44.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:44.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:44.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:45.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:45.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:45.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:45.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:46.074 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:38:46.069+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:46.073275+0000) 2019-10-18T06:38:46.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:46.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:46.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:46.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:47.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:47.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:47.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:47.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:48.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:48.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:48.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:48.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:48.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:48.826+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:49.066 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:38:49.061+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:49.064951+0000) 2019-10-18T06:38:49.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:49.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:49.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:49.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:50.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:50.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:50.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:50.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:51.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:51.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:51.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:51.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:51.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:38:51.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:51.865705+0000) 2019-10-18T06:38:52.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:52.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:52.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:52.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:53.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:53.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:53.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:53.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:53.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:53.830+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:54.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:38:54.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:54.329106+0000) 2019-10-18T06:38:54.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:54.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:54.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:54.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:55.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:55.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:55.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:55.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:56.074 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:38:56.068+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:56.073454+0000) 2019-10-18T06:38:56.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:56.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:56.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:56.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:57.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:57.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:57.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:57.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:58.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:58.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:58.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:58.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:38:58.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:38:58.830+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:38:59.066 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:38:59.060+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:38:59.065120+0000) 2019-10-18T06:38:59.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:59.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:38:59.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:38:59.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:00.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:00.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:00.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:00.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:01.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:01.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:01.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:01.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:01.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:39:01.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:01.865803+0000) 2019-10-18T06:39:02.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:02.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:02.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:02.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:03.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:03.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:03.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:03.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:03.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:03.830+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:04.330 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:39:04.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:04.329210+0000) 2019-10-18T06:39:04.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:04.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:04.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:04.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:05.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:05.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:05.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:05.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:06.075 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:39:06.068+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:06.073637+0000) 2019-10-18T06:39:06.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:06.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:06.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:06.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:07.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:07.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:07.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:07.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:08.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:08.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:08.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:08.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:08.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:08.830+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:09.066 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:39:09.060+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:09.065288+0000) 2019-10-18T06:39:09.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:09.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:09.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:09.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:10.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:10.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:10.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:10.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:11.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:11.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:11.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:11.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:11.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:39:11.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:11.865894+0000) 2019-10-18T06:39:12.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:12.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:12.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:12.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:13.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:13.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:13.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:13.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:13.839 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:13.830+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:14.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:39:14.322+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:14.329317+0000) 2019-10-18T06:39:14.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:14.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:14.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:14.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:15.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:15.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:15.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:15.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:16.076 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:39:16.068+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:16.073822+0000) 2019-10-18T06:39:16.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:16.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:16.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:16.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:17.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:17.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:17.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:17.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:18.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:18.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:18.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:18.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:18.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:18.834+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:19.066 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:39:19.060+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:19.065496+0000) 2019-10-18T06:39:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:19.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:19.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:19.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:20.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:20.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:20.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:20.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:21.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:21.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:21.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:21.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:21.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:39:21.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:21.865991+0000) 2019-10-18T06:39:22.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:22.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:22.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:22.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:23.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:23.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:23.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:23.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:23.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:23.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:24.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:39:24.321+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:24.329426+0000) 2019-10-18T06:39:24.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:24.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:24.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:24.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:25.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:25.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:25.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:25.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:26.075 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:39:26.068+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:26.074003+0000) 2019-10-18T06:39:26.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:26.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:26.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:26.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:27.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:27.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:27.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:27.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:28.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:28.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:28.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:28.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:28.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:28.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:29.067 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:39:29.060+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:29.065690+0000) 2019-10-18T06:39:29.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:29.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:29.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:29.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:30.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:30.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:30.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:30.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:31.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:31.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:31.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:31.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:31.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:39:31.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:31.866092+0000) 2019-10-18T06:39:32.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:32.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:32.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:32.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:33.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:33.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:33.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:33.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:33.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:33.833+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:34.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:39:34.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:34.329541+0000) 2019-10-18T06:39:34.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:34.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:34.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:34.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:35.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:35.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:35.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:35.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:36.075 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:39:36.072+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:36.074209+0000) 2019-10-18T06:39:36.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:36.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:36.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:36.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:37.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:37.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:37.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:37.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:38.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:38.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:38.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:38.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:38.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:38.837+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:39.067 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:39:39.060+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:39.065882+0000) 2019-10-18T06:39:39.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:39.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:39.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:39.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:40.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:40.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:40.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:40.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:41.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:41.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:41.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:41.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:41.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:39:41.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:41.866184+0000) 2019-10-18T06:39:42.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:42.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:42.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:42.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:43.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:43.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:43.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:43.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:43.845 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:43.841+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:44.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:39:44.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:44.329653+0000) 2019-10-18T06:39:44.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:44.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:44.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:44.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:45.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:45.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:45.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:45.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:46.075 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:39:46.072+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:46.074387+0000) 2019-10-18T06:39:46.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:46.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:46.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:46.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:47.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:47.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:47.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:47.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:48.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:48.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:48.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:48.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:48.845 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:48.841+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:49.067 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:39:49.060+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:49.066062+0000) 2019-10-18T06:39:49.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:49.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:49.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:49.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:50.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:50.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:50.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:50.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:51.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:51.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:51.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:51.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:51.867 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:39:51.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:51.866280+0000) 2019-10-18T06:39:52.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:52.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:52.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:52.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:53.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:53.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:53.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:53.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:53.846 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:53.841+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:54.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:39:54.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:54.329764+0000) 2019-10-18T06:39:54.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:54.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:54.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:54.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:55.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:55.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:55.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:55.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:56.075 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:39:56.072+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:56.074567+0000) 2019-10-18T06:39:56.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:56.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:56.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:56.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:57.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:57.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:57.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:57.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:58.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:58.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:58.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:58.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:39:58.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:39:58.840+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:39:59.067 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:39:59.064+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:39:59.066233+0000) 2019-10-18T06:39:59.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:59.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:39:59.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:39:59.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:00.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:00.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:00.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:00.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:01.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:01.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:01.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:01.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:01.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:40:01.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:01.866379+0000) 2019-10-18T06:40:02.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:02.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:02.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:02.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:03.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:03.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:03.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:03.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:03.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:03.844+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:04.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:40:04.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:04.329870+0000) 2019-10-18T06:40:04.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:04.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:04.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:04.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:05.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:05.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:05.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:05.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:06.076 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:40:06.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:06.074745+0000) 2019-10-18T06:40:06.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:06.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:06.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:06.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:07.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:07.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:07.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:07.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:08.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:08.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:08.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:08.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:08.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:08.844+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:09.067 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:40:09.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:09.066428+0000) 2019-10-18T06:40:09.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:09.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:09.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:09.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:10.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:10.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:10.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:10.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:11.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:11.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:11.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:11.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:11.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:40:11.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:11.866479+0000) 2019-10-18T06:40:12.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:12.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:12.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:12.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:13.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:13.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:13.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:13.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:13.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:13.844+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:14.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:40:14.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:14.329978+0000) 2019-10-18T06:40:14.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:14.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:14.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:14.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:15.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:15.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:15.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:15.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:16.076 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:40:16.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:16.074926+0000) 2019-10-18T06:40:16.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:16.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:16.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:16.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:17.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:17.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:17.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:17.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:18.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:18.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:18.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:18.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:18.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:18.844+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:19.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:40:19.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:19.066575+0000) 2019-10-18T06:40:19.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:19.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:19.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:19.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:20.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:20.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:20.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:20.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:21.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:21.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:21.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:21.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:21.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:40:21.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:21.866576+0000) 2019-10-18T06:40:22.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:22.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:22.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:22.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:23.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:23.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:23.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:23.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:23.851 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:23.848+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:24.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:40:24.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:24.330084+0000) 2019-10-18T06:40:24.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:24.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:24.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:24.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:25.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:25.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:25.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:25.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:26.076 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:40:26.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:26.075059+0000) 2019-10-18T06:40:26.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:26.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:26.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:26.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:27.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:27.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:27.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:27.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:28.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:28.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:28.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:28.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:28.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:28.848+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:29.068 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:40:29.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:29.066766+0000) 2019-10-18T06:40:29.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:29.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:29.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:29.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:30.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:30.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:30.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:30.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:31.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:31.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:31.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:31.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:31.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:40:31.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:31.866676+0000) 2019-10-18T06:40:32.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:32.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:32.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:32.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:33.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:33.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:33.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:33.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:33.853 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:33.847+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:34.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:40:34.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:34.330189+0000) 2019-10-18T06:40:34.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:34.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:34.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:34.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:35.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:35.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:35.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:35.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:36.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:40:36.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:36.075193+0000) 2019-10-18T06:40:36.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:36.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:36.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:36.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:37.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:37.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:37.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:37.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:38.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:38.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:38.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:38.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:38.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:38.847+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:39.068 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:40:39.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:39.066937+0000) 2019-10-18T06:40:39.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:39.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:39.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:39.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:40.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:40.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:40.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:40.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:41.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:41.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:41.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:41.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:41.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:40:41.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:41.866774+0000) 2019-10-18T06:40:42.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:42.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:42.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:42.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:43.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:43.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:43.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:43.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:43.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:43.847+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:44.331 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:40:44.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:44.330295+0000) 2019-10-18T06:40:44.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:44.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:44.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:44.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:45.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:45.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:45.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:45.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:46.076 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:40:46.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:46.075325+0000) 2019-10-18T06:40:46.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:46.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:46.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:46.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:47.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:47.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:47.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:47.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:48.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:48.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:48.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:48.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:48.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:48.851+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:49.068 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:40:49.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:49.067140+0000) 2019-10-18T06:40:49.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:49.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:49.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:49.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:50.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:50.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:50.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:50.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:51.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:51.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:51.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:51.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:51.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:40:51.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:51.866873+0000) 2019-10-18T06:40:52.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:52.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:52.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:52.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:53.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:53.695+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:53.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:53.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:53.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:53.851+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:54.332 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:40:54.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:54.330405+0000) 2019-10-18T06:40:54.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:54.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:54.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:54.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:55.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:55.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:55.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:55.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:56.076 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:40:56.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:56.075502+0000) 2019-10-18T06:40:56.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:56.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:56.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:56.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:57.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:57.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:57.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:57.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:58.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:58.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:58.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:58.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:40:58.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:40:58.851+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:40:59.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:40:59.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:40:59.067313+0000) 2019-10-18T06:40:59.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:59.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:40:59.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:40:59.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:00.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:00.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:00.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:00.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:01.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:01.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:01.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:01.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:01.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:41:01.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:01.866970+0000) 2019-10-18T06:41:02.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:02.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:02.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:02.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:03.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:03.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:03.858 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:03.851+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:04.332 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:41:04.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:04.330516+0000) 2019-10-18T06:41:04.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:04.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:04.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:04.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:05.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:05.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:05.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:05.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:06.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:41:06.071+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:06.075698+0000) 2019-10-18T06:41:06.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:06.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:06.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:06.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:07.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:07.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:07.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:07.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:08.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:08.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:08.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:08.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:08.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:08.850+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:09.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:41:09.063+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:09.067542+0000) 2019-10-18T06:41:09.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:09.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:09.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:09.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:10.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:10.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:10.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:10.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:11.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:11.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:11.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:11.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:11.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:41:11.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:11.867080+0000) 2019-10-18T06:41:12.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:12.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:12.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:12.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:13.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:13.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:13.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:13.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:13.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:13.854+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:14.332 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:41:14.326+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:14.330626+0000) 2019-10-18T06:41:14.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:14.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:14.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:14.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:15.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:15.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:15.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:15.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:16.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:41:16.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:16.075838+0000) 2019-10-18T06:41:16.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:16.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:16.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:16.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:17.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:17.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:17.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:17.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:18.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:18.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:18.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:18.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:18.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:18.854+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:19.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:41:19.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:19.067724+0000) 2019-10-18T06:41:19.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:19.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:19.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:19.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:20.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:20.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:20.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:20.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:21.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:21.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:21.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:21.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:21.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:41:21.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:21.867181+0000) 2019-10-18T06:41:22.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:22.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:22.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:22.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:23.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:23.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:23.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:23.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:23.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:23.854+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:24.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:41:24.326+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:24.330740+0000) 2019-10-18T06:41:24.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:24.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:24.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:24.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:25.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:25.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:25.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:25.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:26.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:41:26.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:26.076020+0000) 2019-10-18T06:41:26.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:26.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:26.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:26.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:27.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:27.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:27.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:27.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:28.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:28.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:28.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:28.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:28.864 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:28.858+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:29.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:41:29.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:29.067911+0000) 2019-10-18T06:41:29.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:29.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:29.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:29.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:30.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:30.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:30.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:30.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:31.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:41:31.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:31.867280+0000) 2019-10-18T06:41:31.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:31.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:31.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:31.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:32.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:32.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:33.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:33.858+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:33.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:33.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:33.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:33.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:34.332 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:41:34.326+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:34.330854+0000) 2019-10-18T06:41:34.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:34.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:34.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:34.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:35.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:35.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:35.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:35.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:36.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:41:36.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:36.076201+0000) 2019-10-18T06:41:36.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:36.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:36.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:36.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:37.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:37.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:37.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:37.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:38.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:38.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:38.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:38.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:38.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:38.858+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:39.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:41:39.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:39.068115+0000) 2019-10-18T06:41:39.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:39.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:39.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:39.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:40.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:40.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:40.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:40.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:41.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:41.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:41.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:41.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:41.868 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:41:41.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:41.867379+0000) 2019-10-18T06:41:42.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:42.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:42.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:42.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:43.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:43.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:43.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:43.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:43.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:43.857+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:44.332 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:41:44.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:44.330959+0000) 2019-10-18T06:41:44.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:44.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:44.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:44.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:45.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:45.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:45.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:45.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:46.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:41:46.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:46.076344+0000) 2019-10-18T06:41:46.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:46.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:46.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:46.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:47.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:47.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:47.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:47.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:48.868 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:48.861+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:48.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:48.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:48.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:48.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:49.072 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:41:49.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:49.068317+0000) 2019-10-18T06:41:49.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:49.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:49.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:49.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:50.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:50.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:50.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:50.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:51.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:41:51.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:51.867475+0000) 2019-10-18T06:41:51.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:51.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:51.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:51.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:52.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:52.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:52.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:52.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:53.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:53.861+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:53.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:53.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:53.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:53.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:54.332 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:41:54.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:54.331069+0000) 2019-10-18T06:41:54.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:54.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:54.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:54.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:55.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:55.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:55.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:55.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:56.077 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:41:56.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:56.076516+0000) 2019-10-18T06:41:56.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:56.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:56.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:56.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:57.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:57.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:57.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:57.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:58.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:41:58.865+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:41:59.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:59.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:41:59.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:59.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:41:59.070 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:41:59.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:41:59.068493+0000) 2019-10-18T06:42:00.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:59.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:00.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:41:59.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:01.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:01.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:01.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:01.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:01.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:42:01.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:01.867575+0000) 2019-10-18T06:42:02.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:02.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:02.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:02.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:03.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:03.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:03.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:03.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:03.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:03.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:04.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:04.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:04.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:04.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:04.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:42:04.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:04.331185+0000) 2019-10-18T06:42:05.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:05.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:05.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:05.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:06.078 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:42:06.070+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:06.076696+0000) 2019-10-18T06:42:06.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:06.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:06.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:06.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:07.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:07.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:07.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:07.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:08.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:08.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:08.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:08.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:08.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:08.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:09.070 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:42:09.062+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:09.068667+0000) 2019-10-18T06:42:09.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:09.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:09.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:09.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:10.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:10.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:10.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:10.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:11.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:11.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:11.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:11.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:11.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:42:11.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:11.867673+0000) 2019-10-18T06:42:12.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:12.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:12.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:12.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:13.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:13.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:13.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:13.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:13.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:13.869+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:14.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:14.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:14.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:14.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:14.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:42:14.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:14.331300+0000) 2019-10-18T06:42:15.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:15.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:15.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:15.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:16.079 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:42:16.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:16.076876+0000) 2019-10-18T06:42:16.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:16.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:16.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:16.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:17.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:17.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:17.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:17.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:18.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:18.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:18.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:18.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:18.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:18.872+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:19.070 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:42:19.066+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:19.068836+0000) 2019-10-18T06:42:19.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:19.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:19.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:19.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:20.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:20.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:20.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:20.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:21.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:21.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:21.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:21.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:21.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:42:21.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:21.867767+0000) 2019-10-18T06:42:22.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:22.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:22.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:22.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:23.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:23.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:23.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:23.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:23.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:23.872+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:24.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:24.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:24.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:24.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:24.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:42:24.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:24.331412+0000) 2019-10-18T06:42:25.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:25.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:25.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:25.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:26.078 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:42:26.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:26.077061+0000) 2019-10-18T06:42:26.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:26.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:26.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:26.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:27.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:27.124+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:27.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:27.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:28.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:28.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:28.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:28.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:28.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:28.872+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:29.070 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:42:29.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:29.069006+0000) 2019-10-18T06:42:29.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:29.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:29.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:29.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:30.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:30.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:30.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:30.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:31.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:31.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:31.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:31.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:31.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:42:31.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:31.867863+0000) 2019-10-18T06:42:32.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:32.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:32.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:32.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:33.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:33.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:33.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:33.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:33.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:33.872+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:34.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:34.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:34.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:34.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:34.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:42:34.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:34.331521+0000) 2019-10-18T06:42:35.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:35.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:35.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:35.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:36.078 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:42:36.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:36.077242+0000) 2019-10-18T06:42:36.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:36.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:36.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:36.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:37.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:37.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:37.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:37.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:38.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:38.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:38.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:38.124+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:38.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:38.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:39.071 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:42:39.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:39.069171+0000) 2019-10-18T06:42:39.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:39.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:39.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:39.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:40.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:40.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:40.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:40.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:41.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:41.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:41.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:41.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:41.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:42:41.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:41.867963+0000) 2019-10-18T06:42:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:42.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:42.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:43.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:43.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:43.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:43.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:43.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:43.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:44.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:44.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:44.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:44.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:44.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:42:44.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:44.331630+0000) 2019-10-18T06:42:45.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:45.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:45.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:45.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:46.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:46.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:46.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:46.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:46.078 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:42:46.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:46.077424+0000) 2019-10-18T06:42:47.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:47.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:47.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:47.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:48.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:48.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:48.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:48.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:48.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:48.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:49.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:49.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:49.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:49.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:49.070 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:42:49.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:49.069340+0000) 2019-10-18T06:42:50.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:50.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:50.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:50.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:51.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:51.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:51.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:51.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:51.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:42:51.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:51.868064+0000) 2019-10-18T06:42:51.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:51.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:51.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:51.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:52.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:52.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:52.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:52.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:53.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:53.875+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:53.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:53.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:53.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:53.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:54.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:42:54.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:54.331741+0000) 2019-10-18T06:42:54.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:54.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:54.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:54.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:55.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:55.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:55.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:55.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:56.079 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:42:56.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:56.077566+0000) 2019-10-18T06:42:56.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:56.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:56.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:56.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:57.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:57.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:57.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:57.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:58.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:58.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:58.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:58.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:42:58.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:42:58.879+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:42:59.071 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:42:59.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:42:59.069529+0000) 2019-10-18T06:42:59.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:59.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:42:59.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:42:59.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:00.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:00.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:00.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:00.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:01.869 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:43:01.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:01.868161+0000) 2019-10-18T06:43:01.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:01.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:01.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:01.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:02.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:02.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:02.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:02.839+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:03.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:03.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:03.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:03.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:03.884 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:03.879+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:04.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:43:04.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:04.331848+0000) 2019-10-18T06:43:04.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:04.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:04.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:04.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:05.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:05.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:05.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:05.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:06.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:43:06.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:06.077747+0000) 2019-10-18T06:43:06.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:06.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:06.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:06.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:08.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:08.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:08.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:08.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:08.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:08.879+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:09.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:09.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:32:35.054607+0000 (oldest deadline 2019-10-18T06:32:55.054607+0000) 2019-10-18T06:43:09.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:09.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:09.071 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:43:09.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:09.069713+0000) 2019-10-18T06:43:10.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:10.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:11.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:11.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:11.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:43:11.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:11.868262+0000) 2019-10-18T06:43:11.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:11.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:12.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:12.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:13.887 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:13.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:13.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:13.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:14.333 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:43:14.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:14.332147+0000) 2019-10-18T06:43:14.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:14.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:15.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:15.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:16.079 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:43:16.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:16.077914+0000) 2019-10-18T06:43:16.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:16.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:17.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:17.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:18.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:18.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:18.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:18.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:19.071 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:43:19.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:19.069886+0000) 2019-10-18T06:43:19.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:19.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:21.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:21.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:21.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:43:21.863+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:21.868359+0000) 2019-10-18T06:43:22.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:22.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:23.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:23.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:23.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:23.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:24.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:24.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:24.334 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:43:24.327+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:24.332464+0000) 2019-10-18T06:43:25.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:25.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:26.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:43:26.073+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:26.078107+0000) 2019-10-18T06:43:26.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:26.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:27.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:27.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:28.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:28.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:28.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:28.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:29.071 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:43:29.065+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:29.070060+0000) 2019-10-18T06:43:29.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:29.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:30.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:30.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:30.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:30.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:31.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:31.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:31.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:31.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:31.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:43:31.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:31.868462+0000) 2019-10-18T06:43:32.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:32.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:32.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:32.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:33.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:33.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:33.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:33.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:33.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:33.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:34.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:34.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:34.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:34.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:34.334 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:43:34.326+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:34.332575+0000) 2019-10-18T06:43:35.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:35.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:35.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:35.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:36.079 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:43:36.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:36.078321+0000) 2019-10-18T06:43:36.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:36.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:36.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:36.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:37.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:37.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:37.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:37.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:38.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:38.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:38.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:38.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:38.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:38.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:39.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:43:39.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:39.070246+0000) 2019-10-18T06:43:39.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:39.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:39.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:39.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:40.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:40.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:40.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:40.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:41.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:41.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:41.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:41.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:41.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:43:41.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:41.868554+0000) 2019-10-18T06:43:42.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:42.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:42.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:42.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:43.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:43.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:43.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:43.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:43.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:43.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:44.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:44.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:44.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:44.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:44.334 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:43:44.326+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:44.332690+0000) 2019-10-18T06:43:45.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:45.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:45.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:45.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:46.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:46.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:46.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:46.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:46.079 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:43:46.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:46.078476+0000) 2019-10-18T06:43:47.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:47.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:47.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:47.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:48.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:48.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:48.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:48.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:48.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:48.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:49.071 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:43:49.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:49.070459+0000) 2019-10-18T06:43:49.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:49.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:49.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:49.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:50.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:50.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:50.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:51.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:51.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:51.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:51.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:51.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:43:51.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:51.868655+0000) 2019-10-18T06:43:52.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:52.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:52.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:52.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:53.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:53.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:53.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:53.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:53.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:53.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:54.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:54.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:54.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:54.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:54.334 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:43:54.326+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:54.332794+0000) 2019-10-18T06:43:55.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:55.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:55.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:55.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:56.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:56.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:56.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:56.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:56.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:43:56.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:56.078626+0000) 2019-10-18T06:43:57.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:57.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:57.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:57.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:58.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:58.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:58.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:58.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:58.900 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:43:58.894+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:43:59.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:59.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:43:59.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:43:59.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:43:59.072 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:43:59.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:43:59.070665+0000) 2019-10-18T06:44:00.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:00.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:00.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:00.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:01.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:01.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:01.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:01.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:01.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:44:01.862+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:01.868759+0000) 2019-10-18T06:44:02.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:02.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:02.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:02.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:03.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:03.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:03.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:03.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:03.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:03.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:03.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:03.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:03.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:03.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:04.334 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:44:04.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:04.332898+0000) 2019-10-18T06:44:05.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:05.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:05.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:05.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:06.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:44:06.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:06.078803+0000) 2019-10-18T06:44:06.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:06.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:06.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:06.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:07.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:07.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:07.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:07.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:08.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:08.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:08.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:08.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:08.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:08.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:09.072 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:44:09.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:09.070860+0000) 2019-10-18T06:44:09.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:09.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:09.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:09.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:10.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:10.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:10.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:10.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:11.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:11.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:11.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:11.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:11.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:44:11.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:11.868857+0000) 2019-10-18T06:44:12.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:12.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:12.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:12.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:13.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:13.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:13.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:13.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:13.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:13.893+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:13.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:13.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:14.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:13.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:14.334 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:44:14.325+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:14.333000+0000) 2019-10-18T06:44:14.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:14.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:14.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:14.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:15.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:15.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:15.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:15.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:16.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:44:16.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:16.079017+0000) 2019-10-18T06:44:17.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:17.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:17.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:17.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:17.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:17.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:17.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:17.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:18.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:18.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:19.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:19.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:19.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:19.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:19.072 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:44:19.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:19.071037+0000) 2019-10-18T06:44:20.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:20.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:20.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:20.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:21.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:21.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:21.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:21.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:21.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:44:21.861+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:21.868954+0000) 2019-10-18T06:44:22.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:22.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:22.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:22.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:23.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:23.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:23.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:23.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:23.904 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:23.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:24.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:24.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:24.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:24.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:24.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:44:24.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:24.333109+0000) 2019-10-18T06:44:24.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:24.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:24.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:24.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:25.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:25.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:25.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:25.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:26.083 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:44:26.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:26.079182+0000) 2019-10-18T06:44:27.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:27.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:27.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:27.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:27.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:27.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:27.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:27.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:28.905 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:28.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:29.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:29.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:29.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:29.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:29.072 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:44:29.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:29.071232+0000) 2019-10-18T06:44:30.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:30.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:30.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:30.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:31.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:30.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:31.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:30.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:31.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:44:31.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:31.869059+0000) 2019-10-18T06:44:31.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:31.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:31.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:31.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:33.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:32.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:33.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:32.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:33.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:33.901+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:33.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:33.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:33.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:33.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:34.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:44:34.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:34.333218+0000) 2019-10-18T06:44:34.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:34.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:34.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:34.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:35.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:35.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:35.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:35.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:36.080 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:44:36.076+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:36.079394+0000) 2019-10-18T06:44:36.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:36.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:36.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:36.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:37.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:37.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:37.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:37.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:38.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:38.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:39.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:39.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:39.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:39.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:39.072 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:44:39.068+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:39.071425+0000) 2019-10-18T06:44:39.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:39.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:39.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:39.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:40.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:40.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:40.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:40.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:41.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:44:41.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:41.869162+0000) 2019-10-18T06:44:41.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:41.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:41.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:41.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:42.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:42.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:42.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:42.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:43.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:43.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:43.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:43.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:43.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:43.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:44.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:44:44.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:44.333320+0000) 2019-10-18T06:44:44.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:44.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:44.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:44.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:45.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:45.828+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:45.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:45.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:46.081 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:44:46.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:46.079567+0000) 2019-10-18T06:44:46.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:46.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:46.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:46.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:47.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:47.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:47.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:47.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:48.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:48.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:48.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:48.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:48.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:48.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:49.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:44:49.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:49.071599+0000) 2019-10-18T06:44:49.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:49.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:49.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:49.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:50.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:50.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:50.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:50.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:51.870 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:44:51.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:51.869264+0000) 2019-10-18T06:44:51.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:51.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:51.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:51.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:52.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:52.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:52.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:52.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:53.909 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:53.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:53.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:53.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:53.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:53.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:54.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:44:54.328+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:54.333423+0000) 2019-10-18T06:44:55.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:55.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:55.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:55.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:55.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:55.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:55.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:55.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:56.081 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:44:56.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:56.079746+0000) 2019-10-18T06:44:56.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:56.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:56.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:56.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:57.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:57.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:57.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:57.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:58.913 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:44:58.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:44:58.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:58.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:58.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:58.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:44:59.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:44:59.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:44:59.071796+0000) 2019-10-18T06:44:59.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:59.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:44:59.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:44:59.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:01.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:01.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:01.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:01.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:01.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:45:01.864+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:01.869363+0000) 2019-10-18T06:45:02.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:02.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:02.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:02.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:03.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:03.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:03.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:03.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:03.913 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:03.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:04.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:04.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:04.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:04.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:04.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:45:04.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:04.333538+0000) 2019-10-18T06:45:05.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:05.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:05.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:05.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:06.081 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:45:06.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:06.079957+0000) 2019-10-18T06:45:06.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:06.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:06.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:06.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:07.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:07.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:07.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:07.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:08.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:08.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:08.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:08.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:08.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:08.908+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:09.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:45:09.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:09.071965+0000) 2019-10-18T06:45:09.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:09.096+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:09.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:09.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:10.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:10.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:10.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:10.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:11.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:11.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:11.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:11.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:11.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:45:11.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:11.869465+0000) 2019-10-18T06:45:12.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:12.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:12.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:12.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:13.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:13.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:13.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:13.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:13.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:13.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:14.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:14.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:14.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:14.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:14.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:45:14.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:14.333640+0000) 2019-10-18T06:45:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:15.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:15.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:16.081 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:45:16.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:16.080133+0000) 2019-10-18T06:45:16.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:16.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:16.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:16.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:17.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:17.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:18.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:18.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:18.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:18.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:18.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:18.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:19.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:45:19.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:19.072165+0000) 2019-10-18T06:45:19.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:19.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:19.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:19.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:20.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:20.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:20.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:20.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:21.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:21.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:21.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:21.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:21.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:45:21.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:21.869568+0000) 2019-10-18T06:45:22.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:22.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:22.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:22.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:23.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:23.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:23.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:23.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:23.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:23.911+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:24.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:24.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:24.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:24.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:24.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:45:24.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:24.333744+0000) 2019-10-18T06:45:25.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:25.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:25.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:25.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:26.082 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:45:26.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:26.080292+0000) 2019-10-18T06:45:26.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:26.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:26.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:26.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:27.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:27.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:27.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:27.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:28.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:28.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:28.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:28.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:28.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:28.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:29.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:45:29.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:29.072334+0000) 2019-10-18T06:45:29.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:29.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:29.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:29.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:30.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:30.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:30.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:30.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:31.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:31.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:31.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:31.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:31.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:45:31.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:31.869672+0000) 2019-10-18T06:45:32.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:32.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:32.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:32.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:33.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:33.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:33.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:33.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:33.921 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:33.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:34.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:34.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:34.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:34.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:34.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:45:34.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:34.333849+0000) 2019-10-18T06:45:35.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:35.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:35.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:35.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:36.081 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:45:36.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:36.080456+0000) 2019-10-18T06:45:36.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:36.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:36.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:36.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:37.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:37.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:37.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:37.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:38.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:38.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:38.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:38.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:38.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:38.915+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:39.074 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:45:39.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:39.072549+0000) 2019-10-18T06:45:39.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:39.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:39.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:39.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:40.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:40.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:40.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:41.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:41.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:41.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:41.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:41.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:45:41.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:41.869767+0000) 2019-10-18T06:45:42.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:42.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:42.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:42.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:43.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:43.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:43.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:43.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:43.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:43.919+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:44.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:44.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:44.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:44.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:44.336 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:45:44.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:44.333958+0000) 2019-10-18T06:45:45.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:45.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:45.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:45.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:46.082 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:45:46.075+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:46.080600+0000) 2019-10-18T06:45:46.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:46.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:46.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:46.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:47.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:47.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:47.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:47.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:48.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:48.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:48.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:48.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:48.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:48.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:49.074 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:45:49.067+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:49.072779+0000) 2019-10-18T06:45:49.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:49.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:49.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:49.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:50.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:50.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:50.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:50.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:51.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:51.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:51.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:51.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:51.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:45:51.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:51.869881+0000) 2019-10-18T06:45:52.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:52.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:52.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:52.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:53.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:53.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:53.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:53.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:53.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:53.918+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:54.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:45:54.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:54.334066+0000) 2019-10-18T06:45:54.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:54.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:54.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:54.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:55.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:55.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:55.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:55.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:56.082 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:45:56.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:56.080736+0000) 2019-10-18T06:45:56.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:56.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:56.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:56.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:57.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:57.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:57.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:57.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:58.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:58.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:58.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:58.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:45:58.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:45:58.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:45:59.075 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:45:59.066+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:45:59.072990+0000) 2019-10-18T06:45:59.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:59.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:45:59.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:45:59.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:00.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:00.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:00.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:00.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:01.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:01.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:01.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:01.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:01.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:46:01.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:01.869982+0000) 2019-10-18T06:46:02.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:02.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:02.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:02.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:03.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:03.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:03.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:03.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:03.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:03.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:04.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:04.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:04.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:04.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:04.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:46:04.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:04.334182+0000) 2019-10-18T06:46:05.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:05.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:05.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:05.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:06.082 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:46:06.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:06.080917+0000) 2019-10-18T06:46:06.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:06.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:06.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:06.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:07.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:07.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:07.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:07.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:08.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:08.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:08.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:08.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:08.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:08.922+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:09.074 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:46:09.066+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:09.073182+0000) 2019-10-18T06:46:09.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:09.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:09.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:09.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:10.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:10.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:10.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:10.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:11.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:11.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:11.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:11.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:11.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:46:11.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:11.870088+0000) 2019-10-18T06:46:12.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:12.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:12.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:12.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:13.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:13.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:13.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:13.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:13.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:13.926+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:14.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:14.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:14.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:14.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:14.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:46:14.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:14.334285+0000) 2019-10-18T06:46:15.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:15.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:15.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:15.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:16.083 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:46:16.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:16.081096+0000) 2019-10-18T06:46:16.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:16.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:16.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:16.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:17.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:17.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:17.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:17.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:18.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:18.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:18.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:18.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:18.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:18.926+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:19.074 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:46:19.066+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:19.073416+0000) 2019-10-18T06:46:19.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:19.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:19.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:19.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:20.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:20.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:20.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:20.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:21.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:21.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:21.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:21.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:21.871 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:46:21.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:21.870185+0000) 2019-10-18T06:46:22.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:22.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:22.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:22.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:23.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:23.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:23.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:23.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:23.933 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:23.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:24.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:24.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:24.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:24.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:24.335 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:46:24.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:24.334386+0000) 2019-10-18T06:46:25.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:25.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:25.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:25.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:26.082 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:46:26.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:26.081276+0000) 2019-10-18T06:46:26.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:26.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:26.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:26.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:27.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:27.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:27.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:27.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:28.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:28.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:28.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:28.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:28.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:28.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:29.075 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:46:29.066+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:29.073615+0000) 2019-10-18T06:46:29.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:29.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:29.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:29.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:30.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:30.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:30.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:30.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:31.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:31.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:31.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:31.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:31.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:46:31.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:31.870282+0000) 2019-10-18T06:46:32.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:32.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:32.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:32.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:33.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:33.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:33.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:33.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:33.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:33.925+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:34.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:34.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:34.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:34.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:34.336 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:46:34.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:34.334493+0000) 2019-10-18T06:46:35.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:35.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:35.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:35.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:36.083 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:46:36.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:36.081460+0000) 2019-10-18T06:46:36.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:36.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:36.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:36.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:37.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:37.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:37.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:37.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:38.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:38.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:38.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:38.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:38.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:38.929+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:39.075 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:46:39.066+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:39.073806+0000) 2019-10-18T06:46:39.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:39.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:39.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:39.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:40.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:40.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:40.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:40.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:41.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:41.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:41.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:41.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:41.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:46:41.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:41.870382+0000) 2019-10-18T06:46:42.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:42.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:42.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:42.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:43.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:43.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:43.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:43.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:43.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:43.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:44.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:44.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:44.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:44.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:44.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:46:44.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:44.334603+0000) 2019-10-18T06:46:45.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:45.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:45.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:45.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:46.083 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:46:46.074+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:46.081642+0000) 2019-10-18T06:46:46.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:46.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:46.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:46.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:47.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:47.253+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:47.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:47.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:48.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:48.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:48.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:48.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:48.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:48.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:49.077 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:46:49.070+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:49.074034+0000) 2019-10-18T06:46:49.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:49.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:49.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:49.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:50.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:50.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:50.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:50.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:51.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:51.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:51.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:51.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:51.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:46:51.869+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:51.870487+0000) 2019-10-18T06:46:52.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:52.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:52.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:52.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:53.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:53.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:53.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:53.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:53.938 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:53.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:54.336 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:46:54.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:54.334716+0000) 2019-10-18T06:46:54.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:54.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:54.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:54.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:55.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:55.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:55.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:55.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:56.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:46:56.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:56.081823+0000) 2019-10-18T06:46:56.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:56.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:56.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:56.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:57.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:57.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:57.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:57.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:58.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:58.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:58.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:58.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:46:58.940 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:46:58.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:46:59.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:46:59.074+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:46:59.074254+0000) 2019-10-18T06:46:59.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:59.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:46:59.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:46:59.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:00.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:00.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:00.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:00.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:01.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:01.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:01.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:01.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:01.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:47:01.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:01.870586+0000) 2019-10-18T06:47:02.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:02.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:02.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:02.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:03.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:03.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:03.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:03.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:03.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:03.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:04.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:47:04.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:04.334831+0000) 2019-10-18T06:47:04.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:04.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:04.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:04.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:05.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:05.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:05.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:05.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:06.083 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:47:06.077+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:06.082005+0000) 2019-10-18T06:47:06.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:06.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:06.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:06.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:07.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:07.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:07.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:07.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:08.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:08.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:08.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:08.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:08.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:08.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:09.075 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:47:09.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:09.074473+0000) 2019-10-18T06:47:09.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:09.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:09.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:09.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:10.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:10.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:10.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:10.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:11.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:11.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:11.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:11.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:11.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:47:11.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:11.870687+0000) 2019-10-18T06:47:12.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:12.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:12.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:12.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:13.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:13.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:13.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:13.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:13.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:13.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:14.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:14.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:14.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:14.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:14.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:47:14.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:14.334935+0000) 2019-10-18T06:47:15.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:15.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:15.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:15.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:16.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:47:16.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:16.082215+0000) 2019-10-18T06:47:16.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:16.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:16.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:16.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:17.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:17.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:17.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:17.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:18.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:18.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:18.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:18.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:18.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:18.936+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:19.076 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:47:19.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:19.074699+0000) 2019-10-18T06:47:19.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:19.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:19.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:19.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:20.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:20.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:20.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:20.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:21.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:21.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:21.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:21.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:21.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:47:21.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:21.870793+0000) 2019-10-18T06:47:22.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:22.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:22.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:22.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:23.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:23.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:23.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:23.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:23.940+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:24.336 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:47:24.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:24.335036+0000) 2019-10-18T06:47:24.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:24.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:24.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:24.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:25.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:25.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:25.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:25.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:26.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:47:26.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:26.082399+0000) 2019-10-18T06:47:26.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:26.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:26.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:26.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:27.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:27.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:27.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:27.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:28.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:28.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:28.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:28.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:28.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:28.940+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:29.076 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:47:29.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:29.074910+0000) 2019-10-18T06:47:29.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:29.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:29.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:29.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:30.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:30.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:30.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:30.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:31.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:31.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:31.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:31.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:31.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:47:31.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:31.870895+0000) 2019-10-18T06:47:32.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:32.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:32.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:32.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:33.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:33.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:33.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:33.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:33.945 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:33.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:34.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:47:34.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:34.335139+0000) 2019-10-18T06:47:34.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:34.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:34.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:34.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:35.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:35.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:35.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:35.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:36.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:47:36.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:36.082584+0000) 2019-10-18T06:47:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:36.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:36.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:37.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:37.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:37.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:37.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:38.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:38.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:38.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:38.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:38.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:39.076 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:47:39.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:39.075102+0000) 2019-10-18T06:47:39.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:39.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:39.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:39.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:40.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:40.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:41.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:41.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:41.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:41.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:41.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:47:41.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:41.870991+0000) 2019-10-18T06:47:42.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:42.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:42.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:42.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:43.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:43.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:43.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:43.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:43.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:43.939+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:44.336 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:47:44.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:44.335241+0000) 2019-10-18T06:47:44.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:44.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:44.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:44.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:45.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:45.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:45.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:45.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:46.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:47:46.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:46.082794+0000) 2019-10-18T06:47:46.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:46.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:46.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:46.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:47.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:47.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:47.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:47.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:48.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:48.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:48.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:48.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:48.948 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:48.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:49.076 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:47:49.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:49.075328+0000) 2019-10-18T06:47:49.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:49.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:49.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:49.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:50.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:50.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:50.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:50.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:51.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:51.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:51.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:51.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:51.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:47:51.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:51.871099+0000) 2019-10-18T06:47:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:52.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:52.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:53.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:53.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:53.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:53.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:53.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:53.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:54.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:47:54.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:54.335355+0000) 2019-10-18T06:47:54.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:54.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:54.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:54.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:55.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:55.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:55.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:55.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:56.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:47:56.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:56.083005+0000) 2019-10-18T06:47:56.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:56.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:56.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:56.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:57.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:57.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:57.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:57.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:58.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:58.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:58.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:58.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:47:58.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:47:58.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:47:59.080 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:47:59.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:47:59.075527+0000) 2019-10-18T06:47:59.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:59.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:47:59.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:47:59.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:00.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:00.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:00.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:00.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:01.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:01.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:01.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:01.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:01.872 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:48:01.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:01.871205+0000) 2019-10-18T06:48:02.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:02.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:02.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:02.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:03.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:03.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:03.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:03.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:03.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:03.943+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:04.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:48:04.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:04.335473+0000) 2019-10-18T06:48:04.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:04.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:04.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:04.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:05.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:05.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:05.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:05.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:06.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:48:06.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:06.083190+0000) 2019-10-18T06:48:06.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:06.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:06.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:06.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:07.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:07.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:07.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:07.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:08.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:08.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:08.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:08.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:08.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:08.942+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:09.077 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:48:09.073+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:09.075756+0000) 2019-10-18T06:48:09.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:09.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:09.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:09.498+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:10.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:10.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:10.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:10.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:11.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:11.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:11.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:11.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:11.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:48:11.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:11.871303+0000) 2019-10-18T06:48:12.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:12.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:12.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:12.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:13.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:13.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:13.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:13.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:13.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:13.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:14.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:48:14.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:14.335575+0000) 2019-10-18T06:48:14.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:14.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:14.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:14.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:15.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:15.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:15.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:15.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:16.084 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:48:16.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:16.083381+0000) 2019-10-18T06:48:16.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:16.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:16.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:16.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:17.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:17.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:17.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:17.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:18.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:18.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:18.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:18.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:18.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:18.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:19.077 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:48:19.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:19.075957+0000) 2019-10-18T06:48:19.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:19.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:19.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:19.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:20.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:20.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:20.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:20.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:21.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:21.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:21.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:21.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:21.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:48:21.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:21.871401+0000) 2019-10-18T06:48:22.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:22.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:22.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:22.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:23.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:23.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:23.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:23.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:23.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:23.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:24.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:48:24.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:24.335678+0000) 2019-10-18T06:48:24.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:24.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:24.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:24.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:25.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:25.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:25.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:25.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:26.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:48:26.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:26.083573+0000) 2019-10-18T06:48:26.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:26.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:26.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:26.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:27.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:27.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:27.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:27.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:28.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:28.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:28.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:28.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:28.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:28.946+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:29.077 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:48:29.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:29.076154+0000) 2019-10-18T06:48:29.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:29.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:29.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:29.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:30.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:30.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:30.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:30.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:31.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:31.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:31.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:31.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:31.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:48:31.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:31.871504+0000) 2019-10-18T06:48:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:32.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:32.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:32.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:33.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:33.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:33.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:33.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:33.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:33.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:34.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:48:34.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:34.335774+0000) 2019-10-18T06:48:34.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:34.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:34.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:34.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:35.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:35.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:35.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:35.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:36.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:48:36.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:36.083768+0000) 2019-10-18T06:48:36.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:36.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:36.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:36.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:37.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:37.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:37.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:37.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:38.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:38.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:38.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:38.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:38.950+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:39.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:48:39.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:39.076358+0000) 2019-10-18T06:48:39.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:39.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:39.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:39.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:40.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:40.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:40.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:40.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:41.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:41.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:41.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:41.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:41.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:48:41.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:41.871604+0000) 2019-10-18T06:48:42.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:42.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:42.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:42.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:43.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:43.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:43.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:43.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:43.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:43.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:44.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:48:44.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:44.335872+0000) 2019-10-18T06:48:44.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:44.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:44.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:44.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:45.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:45.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:45.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:45.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:46.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:48:46.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:46.083924+0000) 2019-10-18T06:48:46.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:46.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:46.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:46.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:47.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:47.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:47.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:47.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:48.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:48.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:48.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:48.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:48.957 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:48.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:49.077 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:48:49.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:49.076497+0000) 2019-10-18T06:48:49.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:49.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:49.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:49.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:50.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:50.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:50.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:50.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:51.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:51.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:51.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:51.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:51.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:48:51.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:51.871707+0000) 2019-10-18T06:48:52.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:52.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:52.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:52.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:53.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:53.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:53.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:53.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:53.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:53.949+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:54.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:48:54.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:54.335975+0000) 2019-10-18T06:48:54.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:54.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:54.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:54.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:55.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:55.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:55.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:55.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:56.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:48:56.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:56.084038+0000) 2019-10-18T06:48:56.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:56.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:56.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:56.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:57.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:57.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:57.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:57.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:58.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:58.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:58.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:58.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:48:58.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:48:58.953+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:48:59.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:48:59.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:48:59.076614+0000) 2019-10-18T06:48:59.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:59.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:48:59.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:48:59.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:00.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:00.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:00.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:00.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:01.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:01.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:01.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:01.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:01.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:49:01.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:01.871804+0000) 2019-10-18T06:49:02.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:02.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:02.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:02.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:03.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:03.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:03.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:03.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:03.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:03.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:04.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:49:04.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:04.336084+0000) 2019-10-18T06:49:04.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:04.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:04.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:04.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:05.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:05.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:05.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:05.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:06.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:49:06.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:06.084150+0000) 2019-10-18T06:49:06.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:06.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:06.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:06.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:07.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:07.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:07.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:07.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:08.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:08.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:08.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:08.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:08.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:08.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:09.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:49:09.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:09.076736+0000) 2019-10-18T06:49:09.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:09.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:09.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:09.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:10.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:10.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:10.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:11.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:11.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:11.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:11.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:11.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:49:11.869+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:11.871904+0000) 2019-10-18T06:49:12.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:12.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:12.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:12.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:13.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:13.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:13.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:13.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:13.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:13.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:14.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:49:14.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:14.336200+0000) 2019-10-18T06:49:14.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:14.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:14.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:14.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:15.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:15.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:15.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:15.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:16.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:49:16.080+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:16.084274+0000) 2019-10-18T06:49:16.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:16.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:16.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:16.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:17.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:17.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:17.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:18.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:18.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:18.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:18.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:18.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:18.956+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:19.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:49:19.072+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:19.076856+0000) 2019-10-18T06:49:19.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:19.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:19.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:19.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:20.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:20.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:20.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:20.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:21.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:21.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:21.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:21.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:21.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:49:21.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:21.872006+0000) 2019-10-18T06:49:22.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:22.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:22.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:22.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:23.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:23.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:23.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:23.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:23.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:23.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:24.337 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:49:24.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:24.336305+0000) 2019-10-18T06:49:24.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:24.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:24.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:24.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:25.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:25.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:25.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:25.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:26.085 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:49:26.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:26.084385+0000) 2019-10-18T06:49:26.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:26.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:26.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:26.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:27.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:27.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:27.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:27.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:28.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:28.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:28.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:28.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:28.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:28.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:29.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:49:29.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:29.076961+0000) 2019-10-18T06:49:29.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:29.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:29.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:29.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:30.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:30.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:30.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:30.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:31.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:31.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:31.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:31.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:49:31.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:31.872109+0000) 2019-10-18T06:49:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:32.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:32.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:32.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:33.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:33.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:33.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:33.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:33.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:33.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:34.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:49:34.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:34.336407+0000) 2019-10-18T06:49:34.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:34.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:34.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:34.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:35.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:35.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:35.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:35.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:36.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:49:36.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:36.084501+0000) 2019-10-18T06:49:36.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:36.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:36.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:36.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:37.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:37.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:37.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:37.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:38.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:38.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:38.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:38.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:38.966 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:38.960+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:39.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:49:39.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:39.077091+0000) 2019-10-18T06:49:39.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:39.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:39.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:39.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:40.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:40.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:40.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:40.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:41.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:41.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:41.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:41.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:41.873 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:49:41.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:41.872209+0000) 2019-10-18T06:49:42.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:42.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:42.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:42.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:43.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:43.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:43.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:43.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:43.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:43.964+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:44.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:49:44.332+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:44.336506+0000) 2019-10-18T06:49:44.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:44.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:44.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:44.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:45.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:45.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:45.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:45.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:46.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:49:46.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:46.084613+0000) 2019-10-18T06:49:46.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:46.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:46.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:46.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:47.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:47.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:47.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:47.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:48.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:48.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:48.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:48.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:48.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:48.964+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:49.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:49:49.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:49.077224+0000) 2019-10-18T06:49:49.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:49.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:49.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:49.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:50.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:50.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:50.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:50.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:51.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:51.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:51.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:51.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:49:51.868+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:51.872314+0000) 2019-10-18T06:49:52.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:52.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:52.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:52.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:53.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:53.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:53.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:53.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:53.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:53.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:54.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:49:54.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:54.336614+0000) 2019-10-18T06:49:54.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:54.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:54.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:54.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:55.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:55.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:55.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:55.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:56.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:49:56.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:56.084710+0000) 2019-10-18T06:49:56.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:56.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:56.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:56.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:57.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:57.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:57.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:57.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:58.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:58.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:58.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:58.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:49:58.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:49:58.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:49:59.078 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:49:59.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:49:59.077344+0000) 2019-10-18T06:49:59.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:59.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:49:59.506 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:49:59.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:00.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:00.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:00.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:00.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:01.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:01.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:01.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:01.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:01.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:50:01.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:01.872418+0000) 2019-10-18T06:50:02.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:02.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:02.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:02.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:03.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:03.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:03.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:03.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:03.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:03.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:04.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:50:04.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:04.336729+0000) 2019-10-18T06:50:04.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:04.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:04.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:04.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:05.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:05.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:05.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:05.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:06.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:50:06.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:06.084882+0000) 2019-10-18T06:50:06.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:06.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:06.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:06.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:07.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:07.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:07.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:07.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:08.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:08.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:08.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:08.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:08.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:08.963+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:09.079 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:09.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:09.077504+0000) 2019-10-18T06:50:09.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:09.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:09.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:09.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:10.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:10.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:10.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:10.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:11.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:11.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:11.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:11.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:11.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:50:11.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:11.872514+0000) 2019-10-18T06:50:12.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:12.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:12.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:12.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:13.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:13.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:13.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:13.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:13.972 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:13.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:14.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:50:14.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:14.336832+0000) 2019-10-18T06:50:14.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:14.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:14.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:14.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:15.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:15.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:15.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:15.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:16.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:50:16.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:16.085034+0000) 2019-10-18T06:50:16.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:16.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:16.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:16.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:17.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:17.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:17.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:17.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:18.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:18.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:18.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:18.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:18.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:18.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:19.079 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:19.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:19.077638+0000) 2019-10-18T06:50:19.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:19.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:19.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:19.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:20.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:20.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:20.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:20.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:21.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:21.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:21.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:21.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:21.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:50:21.867+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:21.872615+0000) 2019-10-18T06:50:22.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:22.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:22.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:22.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:23.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:23.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:23.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:23.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:23.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:23.967+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:24.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:50:24.331+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:24.336935+0000) 2019-10-18T06:50:24.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:24.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:24.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:24.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:25.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:25.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:25.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:25.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:26.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:50:26.079+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:26.085142+0000) 2019-10-18T06:50:26.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:26.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:26.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:26.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:27.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:27.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:27.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:27.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:28.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:28.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:28.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:28.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:28.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:28.966+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:29.079 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:29.071+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:29.077826+0000) 2019-10-18T06:50:29.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:29.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:29.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:29.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:30.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:30.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:30.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:30.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:31.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:31.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:31.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:31.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:31.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:50:31.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:31.872736+0000) 2019-10-18T06:50:32.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:32.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:32.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:32.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:33.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:33.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:33.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:33.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:33.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:33.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:34.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:50:34.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:34.337040+0000) 2019-10-18T06:50:34.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:34.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:34.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:34.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:35.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:35.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:35.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:35.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:36.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:50:36.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:36.085253+0000) 2019-10-18T06:50:36.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:36.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:36.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:36.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:37.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:37.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:37.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:37.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:38.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:38.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:38.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:38.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:38.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:38.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:39.079 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:39.070+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:39.077996+0000) 2019-10-18T06:50:39.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:39.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:39.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:39.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:40.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:40.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:40.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:40.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:41.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:41.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:41.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:41.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:41.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:50:41.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:41.872834+0000) 2019-10-18T06:50:42.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:42.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:42.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:42.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:43.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:43.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:43.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:43.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:43.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:43.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:44.338 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:50:44.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:44.337142+0000) 2019-10-18T06:50:44.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:44.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:44.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:44.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:45.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:45.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:45.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:45.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:46.086 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:50:46.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:46.085365+0000) 2019-10-18T06:50:46.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:46.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:46.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:46.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:47.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:47.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:47.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:47.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:48.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:48.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:48.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:48.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:48.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:48.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:49.079 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:49.070+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:49.078144+0000) 2019-10-18T06:50:49.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:49.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:49.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:49.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:50.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:50.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:50.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:50.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:51.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:51.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:51.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:51.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:51.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:50:51.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:51.872940+0000) 2019-10-18T06:50:52.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:52.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:52.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:52.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:53.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:53.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:53.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:53.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:53.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:53.970+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:54.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:50:54.330+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:54.337243+0000) 2019-10-18T06:50:54.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:54.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:54.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:54.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:55.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:55.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:55.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:55.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:56.087 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:50:56.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:56.085474+0000) 2019-10-18T06:50:56.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:56.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:56.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:56.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:57.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:57.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:57.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:57.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:58.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:58.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:58.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:58.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:50:58.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:50:58.974+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:50:59.079 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:59.074+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:59.078303+0000) 2019-10-18T06:50:59.080 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:50:59.074+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:50:59.079094+0000) 2019-10-18T06:50:59.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:59.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:50:59.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:50:59.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:00.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:00.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:00.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:00.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:01.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:01.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:01.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:01.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:01.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:51:01.866+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:01.873039+0000) 2019-10-18T06:51:02.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:02.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:02.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:02.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:03.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:03.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:03.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:03.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:03.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:51:03.574+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:03.578537+0000) 2019-10-18T06:51:03.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:03.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:04.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:51:04.329+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:04.337352+0000) 2019-10-18T06:51:04.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:04.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:04.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:04.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:05.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:05.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:05.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:06.087 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:51:06.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:06.085586+0000) 2019-10-18T06:51:06.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:06.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:06.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:06.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:07.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:07.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:07.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:07.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:08.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:08.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:08.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:08.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:08.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:08.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:09.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:09.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:09.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:09.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:10.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:10.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:10.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:10.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 528 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:11.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:11.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:11.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:11.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:11.874 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:51:11.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:11.873137+0000) 2019-10-18T06:51:12.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:12.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:12.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:12.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:13.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:13.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:13.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:13.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:13.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:51:13.574+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:13.578649+0000) 2019-10-18T06:51:13.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:13.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:14.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:51:14.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:14.337458+0000) 2019-10-18T06:51:14.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:14.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:14.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:14.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:15.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:15.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:15.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:15.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:16.087 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:51:16.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:16.085696+0000) 2019-10-18T06:51:16.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:16.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:16.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:16.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:17.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:17.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:17.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:17.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:18.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:18.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:18.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:18.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:18.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:18.973+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:19.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:19.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:19.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:19.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:20.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:20.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:20.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:20.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:21.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:21.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:21.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:21.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:21.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:51:21.865+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:21.873241+0000) 2019-10-18T06:51:22.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:22.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:22.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:22.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:23.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:51:23.574+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:23.578769+0000) 2019-10-18T06:51:23.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:23.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:23.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:23.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:23.984 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:23.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:24.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:51:24.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:24.337567+0000) 2019-10-18T06:51:24.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:24.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:24.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:24.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:25.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:25.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:25.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:25.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:26.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:51:26.078+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:26.085807+0000) 2019-10-18T06:51:26.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:26.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:26.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:26.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:27.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:27.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:27.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:27.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:28.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:28.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:28.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:28.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:28.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:28.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:29.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:29.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:29.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:29.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:30.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:30.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:30.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:30.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:31.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:31.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:31.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:31.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:31.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:51:31.869+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:31.873345+0000) 2019-10-18T06:51:32.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:32.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:32.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:32.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:33.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:33.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:33.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:33.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:33.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:51:33.578+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:33.578878+0000) 2019-10-18T06:51:33.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:33.981+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:34.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:51:34.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:34.337687+0000) 2019-10-18T06:51:34.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:34.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:34.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:34.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:35.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:35.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:35.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:35.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:36.092 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:51:36.082+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:36.085924+0000) 2019-10-18T06:51:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:36.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:36.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:36.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:37.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:37.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:37.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:37.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:38.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:38.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:38.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:38.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:38.986 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:38.980+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:39.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:39.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:39.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:39.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:40.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:40.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:40.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:40.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:41.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:41.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:41.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:41.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:41.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:51:41.872+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:41.873440+0000) 2019-10-18T06:51:42.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:42.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:42.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:42.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:43.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:43.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:43.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:43.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:43.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:51:43.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:43.578992+0000) 2019-10-18T06:51:43.987 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:43.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:44.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:51:44.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:44.337790+0000) 2019-10-18T06:51:44.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:44.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:44.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:44.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:45.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:45.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:45.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:45.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:46.087 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:51:46.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:46.086025+0000) 2019-10-18T06:51:46.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:46.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:46.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:46.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:47.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:47.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:47.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:47.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:48.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:48.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:48.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:48.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:48.988 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:48.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:49.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:49.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:49.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:49.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:50.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:50.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:50.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:50.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:51.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:51.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:51.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:51.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:51.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:51:51.872+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:51.873555+0000) 2019-10-18T06:51:52.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:52.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:52.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:52.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:53.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:53.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:53.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:53.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:53.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:51:53.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:53.579105+0000) 2019-10-18T06:51:53.989 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:53.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:54.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:51:54.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:54.337894+0000) 2019-10-18T06:51:54.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:54.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:54.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:54.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:55.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:55.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:55.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:55.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:56.087 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:51:56.081+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:51:56.086134+0000) 2019-10-18T06:51:56.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:56.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:56.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:56.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:57.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:57.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:57.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:57.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:58.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:58.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:58.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:58.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:51:58.990 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:51:58.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:51:59.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:59.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:51:59.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:51:59.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:00.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:00.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:00.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:00.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:01.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:01.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:01.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:01.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:01.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:52:01.872+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:01.873654+0000) 2019-10-18T06:52:02.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:02.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:02.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:02.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:03.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:03.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:03.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:03.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:03.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:52:03.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:03.579223+0000) 2019-10-18T06:52:03.991 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:03.984+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:04.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:52:04.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:04.338011+0000) 2019-10-18T06:52:04.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:04.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:04.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:04.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:05.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:05.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:05.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:05.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:06.091 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:52:06.085+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:06.086228+0000) 2019-10-18T06:52:06.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:06.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:06.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:06.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:07.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:07.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:07.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:07.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:08.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:08.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:08.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:08.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:08.992 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:08.988+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:09.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:09.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:09.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:09.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:10.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:10.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:10.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:10.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:11.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:11.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:11.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:11.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:11.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:52:11.872+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:11.873752+0000) 2019-10-18T06:52:12.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:12.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:12.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:12.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:13.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:13.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:13.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:13.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:13.581 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:52:13.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:13.579335+0000) 2019-10-18T06:52:13.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:13.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:14.339 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:52:14.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:14.338113+0000) 2019-10-18T06:52:14.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:14.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:14.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:14.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:15.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:15.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:15.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:15.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:16.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:52:16.085+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:16.086343+0000) 2019-10-18T06:52:16.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:16.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:16.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:16.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:17.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:17.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:17.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:17.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:18.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:18.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:18.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:18.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:18.993 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:18.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:19.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:19.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:19.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:19.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:20.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:20.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:20.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:20.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:21.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:21.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:21.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:21.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:21.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:52:21.871+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:21.873848+0000) 2019-10-18T06:52:22.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:22.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:22.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:22.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:23.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:23.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:23.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:23.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:23.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:52:23.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:23.579473+0000) 2019-10-18T06:52:23.994 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:23.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:24.340 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:52:24.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:24.338213+0000) 2019-10-18T06:52:24.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:24.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:24.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:24.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:25.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:25.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:25.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:25.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:26.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:52:26.085+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:26.086455+0000) 2019-10-18T06:52:26.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:26.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:26.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:26.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:27.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:27.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:27.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:27.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:28.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:28.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:28.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:28.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:28.995 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:28.987+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:29.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:29.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:29.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:29.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:30.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:30.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:30.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:30.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:31.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:31.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:31.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:31.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:31.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:52:31.871+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:31.873948+0000) 2019-10-18T06:52:32.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:32.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:32.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:32.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:33.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:33.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:33.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:33.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:33.581 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:52:33.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:33.579572+0000) 2019-10-18T06:52:33.996 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:33.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:34.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:52:34.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:34.338318+0000) 2019-10-18T06:52:34.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:34.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:34.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:34.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:35.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:35.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:35.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:35.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:36.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:52:36.085+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:36.086554+0000) 2019-10-18T06:52:36.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:36.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:36.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:36.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:37.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:37.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:37.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:37.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:38.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:38.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:38.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:38.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:38.997 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:38.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:39.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:39.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:39.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:39.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:40.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:40.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:40.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:40.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:41.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:41.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:41.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:41.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:41.875 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:52:41.871+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:41.874077+0000) 2019-10-18T06:52:42.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:42.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:42.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:42.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:43.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:43.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:43.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:43.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:43.581 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:52:43.577+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:43.579687+0000) 2019-10-18T06:52:43.998 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:43.991+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:44.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:52:44.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:44.338428+0000) 2019-10-18T06:52:44.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:44.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:44.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:44.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:45.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:45.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:45.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:45.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:46.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:52:46.085+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:46.086678+0000) 2019-10-18T06:52:46.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:46.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:46.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:46.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:47.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:47.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:47.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:47.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:48.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:48.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:48.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:48.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:48.999 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:48.990+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:49.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:49.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:49.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:49.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:50.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:50.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:50.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:50.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:51.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:51.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:51.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:51.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:51.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:52:51.870+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:51.874194+0000) 2019-10-18T06:52:52.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:52.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:52.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:52.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:53.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:53.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:53.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:53.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:53.581 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:52:53.576+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:53.579798+0000) 2019-10-18T06:52:54.001 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:53.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:54.340 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:52:54.334+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:54.338523+0000) 2019-10-18T06:52:54.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:54.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:54.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:54.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:55.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:55.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:55.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:55.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:56.093 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:52:56.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:52:56.086790+0000) 2019-10-18T06:52:56.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:56.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:56.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:56.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:57.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:57.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:57.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:57.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:58.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:58.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:58.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:58.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:52:59.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:52:58.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:52:59.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:59.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:52:59.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:52:59.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:00.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:00.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:00.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:00.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:01.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:01.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:01.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:01.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:01.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:01.870+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:01.874307+0000) 2019-10-18T06:53:02.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:02.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:02.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:02.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:03.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:03.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:03.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:03.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:03.581 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:03.576+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:03.579912+0000) 2019-10-18T06:53:03.582 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:03.576+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:03.580994+0000) 2019-10-18T06:53:04.000 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:03.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:04.340 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:53:04.334+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:04.338622+0000) 2019-10-18T06:53:04.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:04.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:04.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:04.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:05.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:05.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:05.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:05.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:06.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:53:06.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:06.086901+0000) 2019-10-18T06:53:06.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:06.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:06.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:06.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:07.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:07.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:07.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:07.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:08.081 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:08.076+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:08.080139+0000) 2019-10-18T06:53:08.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:08.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:08.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:08.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:09.002 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:08.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:09.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:09.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:09.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:09.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:10.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:10.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:10.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:10.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:11.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:11.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:11.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:11.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:11.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:11.870+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:11.874430+0000) 2019-10-18T06:53:12.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:12.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:12.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:12.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:13.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:13.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:13.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:13.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:14.003 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:13.994+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:14.340 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:53:14.334+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:14.338724+0000) 2019-10-18T06:53:14.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:14.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:14.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:14.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:15.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:15.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:15.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:15.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:16.090 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:53:16.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:16.087014+0000) 2019-10-18T06:53:16.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:16.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:16.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:16.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:17.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:17.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:18.081 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:18.076+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:18.080251+0000) 2019-10-18T06:53:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:18.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:18.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:19.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:18.998+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:19.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:19.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:19.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:19.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:20.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:20.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:20.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:20.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:21.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:21.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:21.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:21.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:21.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:21.870+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:21.874555+0000) 2019-10-18T06:53:22.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:22.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:22.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:22.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:23.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:23.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:23.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:23.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:24.004 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:23.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:24.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:53:24.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:24.338828+0000) 2019-10-18T06:53:24.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:24.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:24.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:24.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:25.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:25.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:25.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:25.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:26.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:53:26.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:26.087129+0000) 2019-10-18T06:53:26.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:26.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:26.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:26.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:27.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:27.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:27.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:27.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:28.081 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:28.076+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:28.080362+0000) 2019-10-18T06:53:28.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:28.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:28.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:28.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:29.005 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:28.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:29.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:29.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:29.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:29.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:30.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:30.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:30.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:30.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:31.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:31.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:31.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:31.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:31.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:31.869+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:31.874668+0000) 2019-10-18T06:53:32.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:32.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:32.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:32.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:33.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:33.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:33.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:33.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:34.006 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:33.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:34.340 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:53:34.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:34.338927+0000) 2019-10-18T06:53:34.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:34.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:34.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:34.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:35.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:35.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:35.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:35.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:36.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:53:36.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:36.087241+0000) 2019-10-18T06:53:36.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:36.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:36.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:36.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:37.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:37.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:37.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:38.082 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:38.076+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:38.080475+0000) 2019-10-18T06:53:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:38.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:38.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:39.007 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:38.997+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:39.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:39.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:39.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:39.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:40.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:40.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:40.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:40.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:41.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:41.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:41.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:41.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:41.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:41.869+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:41.874777+0000) 2019-10-18T06:53:42.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:42.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:42.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:42.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:43.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:43.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:43.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:43.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:44.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:44.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:44.340 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:53:44.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:44.339032+0000) 2019-10-18T06:53:44.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:44.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:44.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:44.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:45.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:45.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:45.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:45.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:46.088 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:53:46.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:46.087359+0000) 2019-10-18T06:53:46.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:46.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:43:09.887341+0000 (oldest deadline 2019-10-18T06:43:29.887341+0000) 2019-10-18T06:53:46.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:46.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:47.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:47.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:48.083 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:48.076+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:48.080589+0000) 2019-10-18T06:53:48.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:48.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:49.008 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:49.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:49.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:49.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:50.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:50.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:51.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:51.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:51.876 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:51.873+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:51.874884+0000) 2019-10-18T06:53:51.878 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:51.873+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:51.875866+0000) 2019-10-18T06:53:52.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:52.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:53.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:53.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:54.010 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:54.005+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:54.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:53:54.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:54.339146+0000) 2019-10-18T06:53:54.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:54.569+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:55.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:55.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:56.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:53:56.084+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:56.087476+0000) 2019-10-18T06:53:56.376 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:53:56.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:56.375070+0000) 2019-10-18T06:53:56.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:56.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:57.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:57.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:58.082 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:53:58.076+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:53:58.080701+0000) 2019-10-18T06:53:58.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:58.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:53:59.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:53:59.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:53:59.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:53:59.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:00.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:00.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:01.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:01.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:02.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:02.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:03.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:03.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:04.011 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:04.004+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:04.342 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:54:04.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:04.339249+0000) 2019-10-18T06:54:04.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:04.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:05.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:05.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:06.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:54:06.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:06.087576+0000) 2019-10-18T06:54:06.377 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:54:06.372+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:06.375168+0000) 2019-10-18T06:54:06.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:06.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:07.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:07.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:07.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:07.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:08.082 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:54:08.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:08.080814+0000) 2019-10-18T06:54:08.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:08.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:08.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:08.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:09.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:09.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:09.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:09.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:09.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:09.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:10.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:10.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:10.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:10.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:11.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:11.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:11.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:11.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:12.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:12.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:13.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:13.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:13.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:13.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:14.012 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:14.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:14.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:54:14.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:14.339353+0000) 2019-10-18T06:54:14.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:14.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:14.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:14.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:15.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:15.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:15.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:15.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:16.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:54:16.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:16.087687+0000) 2019-10-18T06:54:16.377 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:54:16.372+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:16.375269+0000) 2019-10-18T06:54:16.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:16.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:16.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:16.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:17.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:17.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:17.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:17.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:18.082 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:54:18.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:18.080923+0000) 2019-10-18T06:54:18.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:18.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:18.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:18.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:19.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:19.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:19.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:19.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:19.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:19.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:20.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:20.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:20.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:21.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:21.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:21.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:21.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:22.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:22.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:22.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:22.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:23.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:23.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:23.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:23.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:24.014 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:24.008+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:24.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:54:24.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:24.339457+0000) 2019-10-18T06:54:24.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:24.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:24.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:24.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:25.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:25.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:25.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:25.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:26.093 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:54:26.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:26.087801+0000) 2019-10-18T06:54:26.377 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:54:26.372+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:26.375379+0000) 2019-10-18T06:54:26.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:26.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:26.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:26.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:27.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:27.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:27.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:27.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:28.083 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:54:28.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:28.081032+0000) 2019-10-18T06:54:28.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:28.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:28.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:28.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:29.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:29.012+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:29.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:29.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:29.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:29.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:30.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:30.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:30.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:30.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:31.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:31.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:31.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:31.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:32.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:32.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:32.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:32.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:33.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:33.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:33.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:33.795+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:34.016 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:34.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:34.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:54:34.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:34.339558+0000) 2019-10-18T06:54:34.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:34.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:34.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:34.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:35.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:35.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:35.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:35.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:36.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:54:36.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:36.087918+0000) 2019-10-18T06:54:36.377 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:54:36.371+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:36.375480+0000) 2019-10-18T06:54:36.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:36.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:36.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:36.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:37.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:37.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:37.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:37.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:38.082 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:54:38.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:38.081143+0000) 2019-10-18T06:54:38.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:38.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:38.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:38.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:39.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:39.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:39.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:39.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:39.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:39.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:40.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:40.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:40.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:40.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:41.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:41.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:41.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:41.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:42.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:42.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:42.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:42.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:43.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:43.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:43.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:43.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:44.017 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:44.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:44.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:54:44.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:44.339664+0000) 2019-10-18T06:54:44.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:44.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:44.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:44.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:45.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:45.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:45.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:45.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:46.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:54:46.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:46.088076+0000) 2019-10-18T06:54:46.377 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:54:46.371+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:46.375579+0000) 2019-10-18T06:54:46.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:46.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:46.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:46.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:47.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:47.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:47.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:47.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:48.082 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:54:48.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:48.081306+0000) 2019-10-18T06:54:48.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:48.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:48.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:48.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:49.018 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:49.011+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:49.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:49.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:49.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:49.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:50.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:50.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:50.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:50.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:51.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:51.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:51.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:51.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:52.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:52.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:52.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:52.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:53.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:53.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:53.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:53.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:54.019 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:54.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:54.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:54:54.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:54.339775+0000) 2019-10-18T06:54:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:54.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:54.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:54.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:55.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:55.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:55.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:55.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:56.089 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:54:56.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:56.088259+0000) 2019-10-18T06:54:56.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:54:56.371+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:56.375686+0000) 2019-10-18T06:54:56.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:56.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:56.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:56.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:57.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:57.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:57.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:57.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:58.083 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:54:58.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:54:58.081482+0000) 2019-10-18T06:54:58.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:58.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:58.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:58.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:54:59.020 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:54:59.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:54:59.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:59.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:54:59.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:54:59.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:00.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:00.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:00.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:00.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:01.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:01.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:01.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:01.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:02.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:02.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:02.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:02.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:03.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:03.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:03.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:03.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:04.021 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:04.015+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:04.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:55:04.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:04.339891+0000) 2019-10-18T06:55:04.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:04.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:04.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:04.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:05.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:05.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:05.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:05.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:06.090 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:06.083+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:06.088466+0000) 2019-10-18T06:55:06.379 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:55:06.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:06.377517+0000) 2019-10-18T06:55:06.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:06.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:06.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:06.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:07.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:07.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:07.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:07.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:08.083 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:55:08.075+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:08.081664+0000) 2019-10-18T06:55:08.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:08.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:08.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:08.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:09.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:09.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:09.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:09.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:09.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:09.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:10.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:10.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:10.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:10.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:11.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:11.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:11.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:11.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:12.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:12.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:12.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:12.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:13.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:13.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:13.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:13.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:14.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:14.014+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:14.342 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:55:14.334+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:14.339990+0000) 2019-10-18T06:55:14.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:14.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:14.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:14.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:15.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:15.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:15.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:15.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:16.090 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:16.082+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:16.088666+0000) 2019-10-18T06:55:16.091 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:16.082+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:16.089836+0000) 2019-10-18T06:55:16.379 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:55:16.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:16.377613+0000) 2019-10-18T06:55:16.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:16.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:16.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:16.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:17.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:17.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:17.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:17.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:18.083 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:55:18.074+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:18.081850+0000) 2019-10-18T06:55:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:18.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:18.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:18.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:19.023 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:19.018+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:19.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:19.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:19.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:19.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:20.590 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:20.582+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:20.589160+0000) 2019-10-18T06:55:20.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:20.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:20.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:20.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:21.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:21.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:21.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:21.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:22.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:22.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:22.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:22.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:23.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:23.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:23.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:23.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:24.024 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:24.018+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:24.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:55:24.334+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:24.340093+0000) 2019-10-18T06:55:24.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:24.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:24.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:24.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:25.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:25.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:25.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:25.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:26.384 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:55:26.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:26.377712+0000) 2019-10-18T06:55:26.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:26.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:26.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:26.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:27.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:27.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:27.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:27.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:28.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:55:28.074+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:28.082035+0000) 2019-10-18T06:55:28.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:28.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:28.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:28.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:29.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:29.018+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:29.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:29.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:29.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:29.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:30.590 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:30.582+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:30.589348+0000) 2019-10-18T06:55:30.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:30.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:30.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:30.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:31.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:31.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:31.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:31.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:32.922+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:32.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:32.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:33.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:33.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:33.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:34.026 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:34.018+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:34.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:55:34.334+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:34.340195+0000) 2019-10-18T06:55:34.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:34.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:34.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:34.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:36.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:36.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:36.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:36.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:36.379 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:55:36.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:36.377810+0000) 2019-10-18T06:55:37.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:37.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:37.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:37.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:37.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:37.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:37.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:37.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:38.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:55:38.078+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:38.082214+0000) 2019-10-18T06:55:38.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:38.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:38.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:38.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:39.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:39.022+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:39.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:39.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:39.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:39.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:40.591 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:40.582+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:40.589538+0000) 2019-10-18T06:55:40.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:40.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:40.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:40.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:42.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:41.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:42.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:41.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:42.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:42.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:42.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:42.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:43.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:43.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:43.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:43.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:44.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:44.021+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:44.341 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:55:44.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:44.340295+0000) 2019-10-18T06:55:44.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:44.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:44.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:44.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:45.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:45.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:45.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:45.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:46.379 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:55:46.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:46.377911+0000) 2019-10-18T06:55:46.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:46.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:46.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:46.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:47.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:47.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:47.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:47.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:48.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:55:48.078+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:48.082429+0000) 2019-10-18T06:55:48.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:48.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:48.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:48.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:49.028 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:49.021+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:49.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:49.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:49.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:49.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:50.591 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:55:50.582+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:50.589741+0000) 2019-10-18T06:55:51.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:50.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:51.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:50.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:52.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:51.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:52.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:51.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:52.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:52.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:52.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:52.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:53.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:53.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:53.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:53.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:54.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:54.021+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:54.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:55:54.333+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:54.340398+0000) 2019-10-18T06:55:54.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:54.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:54.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:54.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:56.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:56.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:56.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:56.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:56.379 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:55:56.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:56.378012+0000) 2019-10-18T06:55:57.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:56.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:57.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:56.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:58.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:58.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:58.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:58.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:58.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:55:58.078+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:55:58.082582+0000) 2019-10-18T06:55:59.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:59.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:59.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:59.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:55:59.030 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:55:59.021+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:55:59.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:59.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:55:59.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:55:59.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:00.594 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:56:00.582+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:00.589920+0000) 2019-10-18T06:56:00.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:00.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:01.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:00.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:01.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:01.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:01.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:01.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:03.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:03.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:03.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:03.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:03.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:03.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:03.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:03.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:04.032 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:04.029+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:04.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:56:04.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:04.340505+0000) 2019-10-18T06:56:04.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:04.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:04.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:04.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:05.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:05.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:05.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:05.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:06.379 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:56:06.377+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:06.378113+0000) 2019-10-18T06:56:06.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:06.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:06.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:06.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:07.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:07.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:07.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:07.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:08.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:56:08.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:08.082717+0000) 2019-10-18T06:56:08.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:08.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:08.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:08.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:09.033 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:09.029+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:09.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:09.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:09.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:09.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:10.591 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:56:10.586+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:10.590096+0000) 2019-10-18T06:56:10.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:10.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:10.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:10.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:11.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:11.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:11.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:11.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:12.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:12.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:12.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:12.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:13.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:13.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:14.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:14.029+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:14.342 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:56:14.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:14.340626+0000) 2019-10-18T06:56:14.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:14.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:14.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:14.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:15.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:15.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:16.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:56:16.377+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:16.378207+0000) 2019-10-18T06:56:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:16.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:16.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:16.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:17.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:17.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:17.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:17.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:18.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:56:18.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:18.082898+0000) 2019-10-18T06:56:18.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:18.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:18.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:18.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:19.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:19.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:19.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:19.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:19.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:19.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:20.591 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:56:20.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:20.590285+0000) 2019-10-18T06:56:20.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:20.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:20.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:20.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:21.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:21.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:21.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:21.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:22.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:22.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:22.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:22.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:23.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:23.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:23.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:23.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:24.034 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:24.028+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:24.342 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:56:24.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:24.340729+0000) 2019-10-18T06:56:24.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:24.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:24.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:24.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:25.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:25.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:25.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:25.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:26.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:56:26.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:26.378309+0000) 2019-10-18T06:56:26.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:26.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:26.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:26.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:27.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:27.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:27.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:27.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:28.084 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:56:28.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:28.083079+0000) 2019-10-18T06:56:28.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:28.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:28.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:28.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:29.035 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:29.032+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:29.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:29.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:29.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:29.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:30.592 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:56:30.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:30.590471+0000) 2019-10-18T06:56:30.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:30.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:30.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:30.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:31.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:31.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:31.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:31.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:32.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:32.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:32.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:32.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:33.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:33.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:33.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:33.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:34.036 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:34.032+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:34.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:56:34.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:34.340828+0000) 2019-10-18T06:56:34.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:34.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:34.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:34.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:35.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:35.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:35.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:35.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:36.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:56:36.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:36.378407+0000) 2019-10-18T06:56:36.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:36.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:36.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:36.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:37.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:37.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:37.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:37.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:38.085 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:56:38.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:38.083267+0000) 2019-10-18T06:56:38.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:38.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:38.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:38.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:39.037 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:39.032+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:39.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:39.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:39.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:39.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:40.592 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:56:40.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:40.590658+0000) 2019-10-18T06:56:40.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:40.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:40.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:40.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:41.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:41.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:42.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:42.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:42.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:42.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:43.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:43.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:43.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:43.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:44.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:44.032+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:44.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:56:44.336+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:44.340929+0000) 2019-10-18T06:56:44.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:44.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:44.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:44.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:45.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:45.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:45.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:45.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:46.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:56:46.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:46.378505+0000) 2019-10-18T06:56:46.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:46.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:46.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:46.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:47.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:47.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:47.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:47.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:48.085 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:56:48.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:48.083444+0000) 2019-10-18T06:56:48.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:48.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:48.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:48.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:49.038 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:49.032+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:49.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:49.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:49.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:49.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:50.592 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:56:50.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:50.590885+0000) 2019-10-18T06:56:50.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:50.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:50.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:50.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:51.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:51.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:51.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:51.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:52.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:52.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:52.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:52.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:53.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:53.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:53.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:53.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:54.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:54.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:54.342 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:56:54.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:54.341032+0000) 2019-10-18T06:56:54.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:54.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:54.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:54.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:55.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:55.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:55.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:55.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:56.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:56:56.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:56.378605+0000) 2019-10-18T06:56:56.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:56.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:56.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:56.851+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:57.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:57.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:57.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:57.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:58.085 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:56:58.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:56:58.083627+0000) 2019-10-18T06:56:58.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:58.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:58.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:58.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:56:59.040 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:56:59.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:56:59.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:59.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:56:59.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:56:59.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:00.592 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:57:00.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:00.591077+0000) 2019-10-18T06:57:01.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:00.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:01.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:00.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:02.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:02.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:02.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:02.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:03.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:02.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:03.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:02.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:03.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:03.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:03.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:03.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:04.041 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:04.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:04.342 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:57:04.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:04.341140+0000) 2019-10-18T06:57:04.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:04.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:04.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:04.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:06.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:06.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:06.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:06.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:06.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:57:06.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:06.378708+0000) 2019-10-18T06:57:07.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:07.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:07.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:07.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:07.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:07.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:07.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:07.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:08.085 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:57:08.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:08.083805+0000) 2019-10-18T06:57:09.042 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:09.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:09.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:09.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:09.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:10.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:10.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:10.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:10.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:10.593 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:57:10.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:10.591287+0000) 2019-10-18T06:57:11.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:11.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:11.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:11.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:12.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:12.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:12.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:12.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:13.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:13.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:13.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:13.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:14.043 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:14.035+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:14.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:14.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:14.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:14.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:14.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:57:14.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:14.341243+0000) 2019-10-18T06:57:15.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:15.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:15.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:15.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:16.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:16.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:16.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:16.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:16.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:57:16.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:16.378807+0000) 2019-10-18T06:57:17.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:17.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:17.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:17.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:18.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:18.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:18.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:18.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:18.085 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:57:18.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:18.083987+0000) 2019-10-18T06:57:19.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:19.039+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:19.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:19.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:19.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:19.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:20.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:20.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:20.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:20.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:20.593 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:57:20.589+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:20.591498+0000) 2019-10-18T06:57:20.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:20.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:20.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:20.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:22.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:22.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:22.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:22.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:22.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:22.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:22.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:22.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:24.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:23.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:24.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:23.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:24.044 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:24.039+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:24.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:57:24.335+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:24.341355+0000) 2019-10-18T06:57:24.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:24.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:24.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:24.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:25.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:25.955+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:25.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:25.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:26.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:57:26.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:26.378907+0000) 2019-10-18T06:57:26.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:26.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:26.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:26.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:27.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:27.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:27.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:27.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:28.085 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:57:28.081+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:28.084198+0000) 2019-10-18T06:57:28.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:28.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:28.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:28.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:29.045 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:29.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:29.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:29.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:29.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:29.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:30.593 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:57:30.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:30.591642+0000) 2019-10-18T06:57:30.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:30.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:30.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:30.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:31.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:31.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:31.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:31.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:32.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:32.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:32.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:32.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:33.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:33.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:33.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:33.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:34.046 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:34.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:34.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:57:34.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:34.341473+0000) 2019-10-18T06:57:34.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:34.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:34.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:34.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:35.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:35.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:35.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:35.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:36.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:57:36.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:36.379009+0000) 2019-10-18T06:57:36.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:36.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:36.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:36.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:37.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:37.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:37.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:37.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:38.086 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:57:38.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:38.084392+0000) 2019-10-18T06:57:38.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:38.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:38.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:38.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:39.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:39.038+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:39.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:39.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:39.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:39.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:40.593 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:57:40.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:40.591823+0000) 2019-10-18T06:57:40.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:40.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:40.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:40.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:41.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:41.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:41.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:41.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:42.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:42.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:42.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:42.906+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:43.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:43.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:43.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:43.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:44.047 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:44.042+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:44.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:57:44.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:44.341570+0000) 2019-10-18T06:57:44.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:44.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:44.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:44.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:45.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:45.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:45.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:45.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:46.384 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:57:46.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:46.379110+0000) 2019-10-18T06:57:46.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:46.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:46.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:46.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:47.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:47.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:47.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:47.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:48.086 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:57:48.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:48.084606+0000) 2019-10-18T06:57:48.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:48.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:48.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:48.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:49.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:49.042+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:49.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:49.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:49.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:49.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:50.593 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:57:50.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:50.592012+0000) 2019-10-18T06:57:50.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:50.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:50.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:50.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:52.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:51.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:52.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:51.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:53.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:53.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:53.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:53.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:53.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:53.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:53.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:53.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:54.049 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:54.042+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:54.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:57:54.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:54.341668+0000) 2019-10-18T06:57:54.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:54.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:54.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:54.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:55.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:55.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:55.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:55.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:56.380 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:57:56.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:56.379212+0000) 2019-10-18T06:57:56.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:56.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:56.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:56.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:57.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:57.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:57.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:57.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:58.086 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:57:58.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:57:58.084776+0000) 2019-10-18T06:57:58.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:58.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:58.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:58.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:57:59.050 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:57:59.042+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:57:59.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:59.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:57:59.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:57:59.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:00.594 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:58:00.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:00.592198+0000) 2019-10-18T06:58:00.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:00.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:00.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:00.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:01.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:01.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:01.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:01.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:02.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:02.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:02.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:02.926+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:03.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:03.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:03.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:03.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:04.051 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:04.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:04.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:58:04.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:04.341766+0000) 2019-10-18T06:58:04.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:04.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:04.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:04.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:05.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:05.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:05.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:05.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:06.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:58:06.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:06.379320+0000) 2019-10-18T06:58:06.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:06.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:06.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:06.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:07.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:07.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:07.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:07.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:08.086 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:58:08.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:08.084979+0000) 2019-10-18T06:58:08.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:08.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:08.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:08.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:09.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:09.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:09.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:09.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:09.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:09.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:10.595 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:58:10.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:10.592412+0000) 2019-10-18T06:58:10.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:10.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:10.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:10.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:11.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:11.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:11.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:11.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:12.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:12.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:12.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:12.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:13.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:13.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:13.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:13.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:14.052 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:14.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:14.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:58:14.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:14.341872+0000) 2019-10-18T06:58:14.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:14.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:14.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:14.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:15.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:15.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:15.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:15.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:16.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:58:16.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:16.379422+0000) 2019-10-18T06:58:16.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:16.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:16.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:16.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:17.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:17.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:17.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:17.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:18.090 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:58:18.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:18.085170+0000) 2019-10-18T06:58:18.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:18.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:18.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:18.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:19.053 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:19.045+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:19.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:19.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:19.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:19.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:20.594 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:58:20.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:20.592597+0000) 2019-10-18T06:58:20.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:20.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:20.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:20.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:21.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:21.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:21.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:21.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:22.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:22.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:22.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:22.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:23.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:23.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:23.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:23.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:24.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:24.049+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:24.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:58:24.341+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:24.341981+0000) 2019-10-18T06:58:24.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:24.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:24.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:24.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:25.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:25.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:25.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:25.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:26.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:58:26.377+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:26.379524+0000) 2019-10-18T06:58:26.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:26.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:26.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:26.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:27.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:27.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:27.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:27.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:28.086 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:58:28.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:28.085355+0000) 2019-10-18T06:58:28.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:28.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:28.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:28.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:29.055 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:29.053+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:29.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:29.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:29.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:29.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:30.594 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:58:30.588+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:30.592786+0000) 2019-10-18T06:58:30.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:30.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:30.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:30.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:31.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:31.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:31.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:31.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:32.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:32.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:32.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:32.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:33.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:33.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:33.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:33.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:34.056 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:34.053+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:34.343 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:58:34.341+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:34.342093+0000) 2019-10-18T06:58:34.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:34.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:34.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:34.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:35.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:35.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:35.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:35.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:36.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:58:36.377+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:36.379623+0000) 2019-10-18T06:58:36.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:36.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:36.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:36.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:38.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:37.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:38.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:38.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:38.087 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:58:38.080+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:38.085548+0000) 2019-10-18T06:58:39.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:39.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:39.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:39.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:39.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:39.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:40.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:40.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:40.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:40.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:40.594 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:58:40.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:40.592966+0000) 2019-10-18T06:58:41.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:41.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:41.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:41.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:42.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:42.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:42.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:42.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:42.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:42.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:42.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:42.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:44.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:44.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:44.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:44.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:44.057 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:44.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:44.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:58:44.340+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:44.342197+0000) 2019-10-18T06:58:45.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:45.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:45.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:45.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:46.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:46.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:46.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:46.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:46.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:58:46.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:46.379719+0000) 2019-10-18T06:58:47.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:47.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:47.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:47.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:48.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:48.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:48.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:48.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:48.087 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:58:48.079+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:48.085734+0000) 2019-10-18T06:58:49.058 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:49.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:49.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:49.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:49.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:49.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:50.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:50.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:50.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:50.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:58:50.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:50.593157+0000) 2019-10-18T06:58:51.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:51.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:51.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:51.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:52.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:52.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:52.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:52.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:53.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:53.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:53.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:53.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:54.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:54.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:54.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:54.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:54.059 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:54.052+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:54.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:58:54.340+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:54.342298+0000) 2019-10-18T06:58:55.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:55.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:55.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:56.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:56.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:56.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:56.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:56.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:58:56.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:56.379820+0000) 2019-10-18T06:58:57.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:57.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:57.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:57.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:58.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:58.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:58.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:58.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:58:58.087 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:58:58.079+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:58:58.085915+0000) 2019-10-18T06:58:59.061 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:58:59.056+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:58:59.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:59.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:58:59.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:58:59.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:00.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:00.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:00.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:00.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:00.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:59:00.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:00.593339+0000) 2019-10-18T06:59:01.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:01.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:01.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:01.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:02.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:02.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:02.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:02.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:03.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:03.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:03.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:03.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:04.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:04.056+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:04.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:04.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:04.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:04.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:04.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:59:04.340+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:04.342402+0000) 2019-10-18T06:59:05.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:05.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:05.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:05.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:06.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:06.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:06.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:06.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:06.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:59:06.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:06.379939+0000) 2019-10-18T06:59:07.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:07.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:07.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:07.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:08.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:08.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:08.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:08.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:08.087 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:59:08.079+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:08.086103+0000) 2019-10-18T06:59:09.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:09.056+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:09.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:09.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:09.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:09.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:10.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:10.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:10.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:10.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:10.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:59:10.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:10.593524+0000) 2019-10-18T06:59:11.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:11.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:11.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:11.160+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:12.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:12.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:12.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:12.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:13.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:13.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:13.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:13.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:14.062 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:14.055+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:14.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:14.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:14.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:14.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:14.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:59:14.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:14.342510+0000) 2019-10-18T06:59:15.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:15.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:15.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:15.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:16.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:16.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:16.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:16.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:16.387 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:59:16.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:16.380037+0000) 2019-10-18T06:59:17.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:17.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:17.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:17.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:18.087 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:59:18.083+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:18.086287+0000) 2019-10-18T06:59:18.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:18.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:18.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:18.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:19.063 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:19.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:19.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:19.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:19.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:19.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:20.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:20.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:20.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:20.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:20.595 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:59:20.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:20.593713+0000) 2019-10-18T06:59:21.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:21.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:21.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:21.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:22.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:22.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:22.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:22.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:23.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:23.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:23.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:23.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:24.064 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:24.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:24.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:24.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:24.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:24.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:24.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:59:24.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:24.342619+0000) 2019-10-18T06:59:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:25.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:25.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:25.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:26.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:26.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:26.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:26.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:26.381 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:59:26.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:26.380139+0000) 2019-10-18T06:59:27.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:27.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:27.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:27.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:28.088 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:59:28.083+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:28.086463+0000) 2019-10-18T06:59:28.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:28.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:28.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:28.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:29.065 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:29.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:29.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:29.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:29.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:29.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:30.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:30.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:30.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:30.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:30.595 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:59:30.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:30.593922+0000) 2019-10-18T06:59:31.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:31.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:31.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:31.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:32.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:32.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:32.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:32.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:33.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:33.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:33.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:33.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:34.066 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:34.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:34.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:34.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:34.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:34.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:34.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:59:34.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:34.342730+0000) 2019-10-18T06:59:35.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:35.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:35.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:35.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:36.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:36.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:36.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:36.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:36.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:59:36.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:36.380245+0000) 2019-10-18T06:59:37.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:37.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:37.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:37.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:38.088 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:59:38.083+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:38.086654+0000) 2019-10-18T06:59:38.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:38.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:38.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:38.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:39.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:39.059+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:39.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:39.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:39.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:39.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:40.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:40.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:40.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:40.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:40.595 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:59:40.587+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:40.594116+0000) 2019-10-18T06:59:41.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:41.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:41.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:41.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:42.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:42.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:42.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:42.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:43.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:43.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:43.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:43.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:44.067 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:44.063+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:44.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:44.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:44.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:44.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:44.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:59:44.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:44.342833+0000) 2019-10-18T06:59:45.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:45.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:45.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:45.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:46.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:46.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:46.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:46.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:46.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:59:46.375+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:46.380345+0000) 2019-10-18T06:59:47.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:47.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:47.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:47.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:48.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:48.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:48.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:48.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:48.088 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:59:48.083+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:48.086836+0000) 2019-10-18T06:59:49.068 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:49.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:49.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:49.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:49.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:49.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:50.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:50.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:50.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:50.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:50.595 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T06:59:50.590+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:50.594324+0000) 2019-10-18T06:59:51.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:51.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:51.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:51.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:52.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:52.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:52.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:52.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:53.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:53.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:53.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:53.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:54.069 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:54.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:54.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:54.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:54.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:54.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:54.344 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T06:59:54.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:54.342911+0000) 2019-10-18T06:59:55.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:55.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:55.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:55.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:56.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:56.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:56.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:56.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:56.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T06:59:56.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:56.380449+0000) 2019-10-18T06:59:57.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:57.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:57.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:57.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:58.088 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T06:59:58.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T05:59:58.087014+0000) 2019-10-18T06:59:58.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:58.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:58.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:58.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T06:59:59.070 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T06:59:59.062+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T06:59:59.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:59.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T06:59:59.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T06:59:59.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:00.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:00.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:00.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:00.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:00.596 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:00:00.590+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:00.594518+0000) 2019-10-18T07:00:01.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:01.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:01.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:01.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:02.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:02.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:02.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:02.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:03.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:03.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:03.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:03.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:04.071 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:04.066+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:04.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:04.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:04.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:04.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:04.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:00:04.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:04.343015+0000) 2019-10-18T07:00:05.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:05.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:05.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:05.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:06.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:06.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:06.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:06.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:06.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:00:06.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:06.380560+0000) 2019-10-18T07:00:07.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:07.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:07.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:07.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:08.089 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:00:08.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:08.087193+0000) 2019-10-18T07:00:08.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:08.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:08.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:08.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:09.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:09.066+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:09.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:09.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:09.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:09.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:10.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:10.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:10.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:10.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:10.596 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:00:10.590+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:10.594699+0000) 2019-10-18T07:00:11.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:11.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:11.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:11.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:12.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:12.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:12.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:12.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:13.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:13.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:13.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:13.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:14.072 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:14.066+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:14.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:14.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:14.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:14.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:14.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:00:14.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:14.343122+0000) 2019-10-18T07:00:15.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:15.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:15.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:15.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:16.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:16.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:16.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:16.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:16.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:00:16.374+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:16.380656+0000) 2019-10-18T07:00:17.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:17.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:17.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:17.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:18.089 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:00:18.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:18.087375+0000) 2019-10-18T07:00:18.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:18.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:18.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:18.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:19.073 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:19.066+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:19.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:19.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:19.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:19.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:20.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:20.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:20.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:20.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:20.596 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:00:20.590+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:20.594880+0000) 2019-10-18T07:00:21.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:21.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:21.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:21.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:22.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:22.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:22.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:22.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:23.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:23.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:23.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:23.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:24.074 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:24.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:24.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:24.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:24.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:24.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:24.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:00:24.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:24.343224+0000) 2019-10-18T07:00:25.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:25.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:25.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:25.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:26.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:26.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:26.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:26.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:00:26.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:26.380758+0000) 2019-10-18T07:00:27.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:27.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:27.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:27.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:28.089 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:00:28.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:28.087555+0000) 2019-10-18T07:00:28.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:28.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:28.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:28.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:29.075 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:29.065+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:29.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:29.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:29.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:29.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:30.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:30.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:30.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:30.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:30.596 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:00:30.590+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:30.595065+0000) 2019-10-18T07:00:31.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:31.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:31.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:31.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:32.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:32.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:32.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:32.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:33.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:33.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:34.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:34.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:34.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:34.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:34.077 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:34.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:34.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:00:34.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:34.343327+0000) 2019-10-18T07:00:35.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:35.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:35.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:35.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:36.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:36.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:36.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:36.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:36.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:00:36.373+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:36.380856+0000) 2019-10-18T07:00:37.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:37.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:37.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:37.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:38.089 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:00:38.082+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:38.087739+0000) 2019-10-18T07:00:38.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:38.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:38.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:38.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:39.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:39.069+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:39.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:39.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:39.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:39.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:40.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:40.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:40.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:40.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:40.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:00:40.590+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:40.595245+0000) 2019-10-18T07:00:41.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:41.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:41.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:41.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:42.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:42.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:42.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:42.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:43.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:43.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:43.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:43.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:44.078 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:44.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:44.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:44.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:44.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:44.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:44.347 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:00:44.341+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:44.343423+0000) 2019-10-18T07:00:45.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:45.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:45.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:45.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:46.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:46.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:46.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:46.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:46.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:00:46.377+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:46.380955+0000) 2019-10-18T07:00:47.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:47.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:47.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:47.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:48.089 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:00:48.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:48.087920+0000) 2019-10-18T07:00:48.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:48.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:48.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:48.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:49.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:49.073+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:49.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:49.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:49.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:49.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:50.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:50.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:50.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:50.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:50.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:00:50.594+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:50.595432+0000) 2019-10-18T07:00:51.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:51.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:51.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:51.253+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:52.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:52.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:52.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:52.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:53.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:53.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:53.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:53.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:54.079 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:54.077+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:54.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:54.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:54.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:54.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:54.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:00:54.341+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:54.343527+0000) 2019-10-18T07:00:55.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:55.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:55.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:55.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:56.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:56.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:56.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:56.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:56.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:00:56.377+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:56.381057+0000) 2019-10-18T07:00:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:57.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:57.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:57.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:58.089 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:00:58.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:00:58.088101+0000) 2019-10-18T07:00:58.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:58.317+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:58.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:58.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:00:59.080 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:00:59.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:00:59.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:59.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:00:59.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:00:59.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:00.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:00.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:00.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:00.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:00.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:01:00.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:00.595616+0000) 2019-10-18T07:01:01.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:01.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:01.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:01.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:02.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:02.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:02.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:02.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:03.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:03.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:04.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:04.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:04.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:04.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:04.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:04.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:04.347 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:01:04.340+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:04.343639+0000) 2019-10-18T07:01:05.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:05.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:05.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:05.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:06.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:06.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:06.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:06.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:06.382 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:01:06.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:06.381161+0000) 2019-10-18T07:01:07.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:07.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:07.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:07.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:08.092 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:01:08.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:08.088314+0000) 2019-10-18T07:01:08.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:08.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:08.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:08.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:09.082 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:09.076+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:09.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:09.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:09.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:09.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:10.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:10.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:10.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:10.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:10.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:01:10.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:10.595828+0000) 2019-10-18T07:01:11.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:11.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:11.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:11.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:12.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:12.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:12.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:12.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:13.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:13.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:13.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:13.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:14.083 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:14.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:14.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:14.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:14.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:14.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:14.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:01:14.340+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:14.343753+0000) 2019-10-18T07:01:15.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:15.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:15.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:15.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:16.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:16.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:16.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:16.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:16.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:01:16.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:16.381262+0000) 2019-10-18T07:01:17.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:17.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:17.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:17.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:18.090 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:01:18.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:18.088497+0000) 2019-10-18T07:01:18.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:18.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:18.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:18.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:19.085 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:19.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:19.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:19.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:19.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:19.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:20.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:20.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:20.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:20.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:20.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:01:20.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:20.596009+0000) 2019-10-18T07:01:21.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:21.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:21.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:21.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:22.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:22.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:22.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:22.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:23.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:23.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:23.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:23.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:24.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:24.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:24.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:24.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:24.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:24.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:24.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:01:24.340+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:24.343868+0000) 2019-10-18T07:01:25.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:25.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:25.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:25.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:26.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:26.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:26.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:26.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:26.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:01:26.376+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:26.381362+0000) 2019-10-18T07:01:27.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:27.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:27.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:27.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:28.095 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:01:28.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:28.088681+0000) 2019-10-18T07:01:28.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:28.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:28.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:28.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:29.086 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:29.080+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:29.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:29.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:29.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:29.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:30.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:30.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:30.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:30.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:30.597 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:01:30.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:30.596197+0000) 2019-10-18T07:01:31.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:31.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:31.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:31.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:32.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:32.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:32.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:32.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:33.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:33.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:33.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:33.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:34.087 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:34.079+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:34.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:01:34.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:34.343969+0000) 2019-10-18T07:01:34.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:34.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:34.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:34.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:35.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:35.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:35.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:35.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:36.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:01:36.379+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:36.381465+0000) 2019-10-18T07:01:36.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:36.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:36.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:36.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:37.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:37.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:37.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:37.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:38.090 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:01:38.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:38.088873+0000) 2019-10-18T07:01:38.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:38.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:38.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:38.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:39.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:39.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:39.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:39.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:39.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:39.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:40.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:40.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:40.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:40.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:40.598 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:01:40.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:40.596389+0000) 2019-10-18T07:01:41.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:41.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:41.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:41.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:42.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:42.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:42.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:42.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:43.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:43.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:43.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:43.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:44.088 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:44.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:44.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:01:44.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:44.344072+0000) 2019-10-18T07:01:44.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:44.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:44.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:44.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:45.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:45.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:45.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:45.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:46.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:01:46.379+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:46.381561+0000) 2019-10-18T07:01:46.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:46.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:46.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:46.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:47.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:47.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:47.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:47.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:48.090 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:01:48.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:48.089055+0000) 2019-10-18T07:01:48.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:48.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:48.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:48.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:49.089 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:49.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:49.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:49.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:50.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:50.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:50.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:50.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:50.598 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:01:50.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:50.596573+0000) 2019-10-18T07:01:51.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:51.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:51.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:51.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:52.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:52.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:52.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:52.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:53.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:53.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:53.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:53.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:54.090 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:54.083+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:54.345 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:01:54.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:54.344177+0000) 2019-10-18T07:01:54.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:54.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:54.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:54.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:55.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:55.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:55.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:55.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:56.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:01:56.379+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:56.381659+0000) 2019-10-18T07:01:56.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:56.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:56.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:56.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:57.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:57.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:57.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:57.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:58.090 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:01:58.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:01:58.089235+0000) 2019-10-18T07:01:58.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:58.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:58.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:58.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:01:59.091 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:01:59.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:01:59.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:59.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:01:59.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:01:59.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:00.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:00.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:00.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:00.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:00.598 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:00.593+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:00.596758+0000) 2019-10-18T07:02:01.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:01.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:01.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:01.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:02.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:02.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:02.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:02.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:03.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:03.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:03.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:03.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:04.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:04.087+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:04.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:04.339+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:04.344277+0000) 2019-10-18T07:02:04.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:04.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:04.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:04.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:05.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:05.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:05.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:05.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:06.383 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:06.379+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:06.381754+0000) 2019-10-18T07:02:06.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:06.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:06.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:06.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:07.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:07.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:07.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:07.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:08.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:02:08.085+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:08.089374+0000) 2019-10-18T07:02:08.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:08.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:08.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:08.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:09.093 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:09.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:09.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:09.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:09.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:09.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:10.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:10.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:10.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:10.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:10.606 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:10.592+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:10.596908+0000) 2019-10-18T07:02:11.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:11.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:11.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:11.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:12.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:12.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:12.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:12.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:13.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:13.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:13.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:13.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:14.094 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:14.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:14.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:14.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:14.344380+0000) 2019-10-18T07:02:14.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:14.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:14.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:14.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:15.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:15.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:15.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:15.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:16.384 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:16.378+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:16.381853+0000) 2019-10-18T07:02:16.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:16.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:16.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:16.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:17.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:17.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:17.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:17.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:18.091 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:02:18.084+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:18.089566+0000) 2019-10-18T07:02:18.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:18.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:18.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:18.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:19.101 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:19.086+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:19.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:19.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:19.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:19.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:20.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:20.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:20.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:20.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:20.598 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:20.592+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:20.597097+0000) 2019-10-18T07:02:21.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:21.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:21.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:21.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:22.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:22.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:22.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:22.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:23.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:23.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:23.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:23.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:24.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:24.090+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:24.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:24.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:24.344490+0000) 2019-10-18T07:02:24.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:24.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:24.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:24.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:25.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:25.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:25.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:25.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:26.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:26.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:26.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:26.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:26.384 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:26.378+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:26.381949+0000) 2019-10-18T07:02:26.385 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:26.378+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:26.383644+0000) 2019-10-18T07:02:27.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:27.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:27.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:27.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:28.091 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:02:28.084+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:28.089745+0000) 2019-10-18T07:02:28.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:28.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:28.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:28.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:29.096 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:29.090+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:29.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:29.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:29.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:29.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:30.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:30.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:30.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:30.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:30.598 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:30.592+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:30.597279+0000) 2019-10-18T07:02:30.883 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:30.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:30.882155+0000) 2019-10-18T07:02:31.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:31.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:31.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:31.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:32.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:32.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:32.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:32.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:33.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:33.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:33.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:33.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:34.097 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:34.090+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:34.346 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:34.338+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:34.344603+0000) 2019-10-18T07:02:34.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:34.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:34.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:34.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:35.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:35.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:35.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:35.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:36.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:36.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:36.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:36.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:37.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:37.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:37.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:37.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:38.092 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:02:38.084+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:38.089889+0000) 2019-10-18T07:02:38.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:38.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:38.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:38.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:39.098 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:39.090+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:39.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:39.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:39.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:39.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:40.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:40.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:40.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:40.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:40.599 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:40.592+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:40.597456+0000) 2019-10-18T07:02:40.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:40.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:40.882260+0000) 2019-10-18T07:02:41.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:41.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:41.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:41.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:42.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:42.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:42.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:42.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:43.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:43.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:43.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:43.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:44.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:44.089+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:44.347 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:44.337+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:44.344702+0000) 2019-10-18T07:02:44.348 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:44.341+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:44.345438+0000) 2019-10-18T07:02:44.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:44.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:44.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:44.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:45.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:45.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:45.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:45.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:46.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:46.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:46.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:46.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:47.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:47.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:47.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:47.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:48.092 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:02:48.084+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:48.090071+0000) 2019-10-18T07:02:48.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:48.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:48.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:48.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:48.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:48.837+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:48.844912+0000) 2019-10-18T07:02:49.099 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:49.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:49.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:49.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:49.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:49.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:50.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:50.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:50.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:50.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:50.599 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:50.592+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:50.597661+0000) 2019-10-18T07:02:50.600 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:50.596+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:50.598613+0000) 2019-10-18T07:02:50.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:02:50.877+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:50.882368+0000) 2019-10-18T07:02:51.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:51.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:51.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:51.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:52.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:52.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:52.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:52.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:53.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:53.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:53.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:53.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:54.100 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:54.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:54.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:54.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:54.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:54.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:55.100 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:02:55.092+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:55.098155+0000) 2019-10-18T07:02:55.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:55.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:55.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:55.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:56.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:56.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:56.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:56.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:57.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:57.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:57.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:57.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:58.093 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:02:58.088+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:58.090251+0000) 2019-10-18T07:02:58.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:58.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:58.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:58.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:02:58.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:02:58.837+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:02:58.845016+0000) 2019-10-18T07:02:59.102 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:02:59.093+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:02:59.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:59.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:02:59.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:02:59.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:00.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:00.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:00.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:00.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:00.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:03:00.877+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:00.882473+0000) 2019-10-18T07:03:01.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:01.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:01.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:01.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:02.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:02.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:02.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:02.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:03.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:03.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:03.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:03.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:04.103 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:04.097+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:04.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:04.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:04.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:04.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:05.100 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:03:05.096+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:05.098358+0000) 2019-10-18T07:03:05.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:05.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:05.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:05.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:06.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:06.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:06.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:06.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:07.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:07.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:07.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:07.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:08.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:03:08.088+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:08.090458+0000) 2019-10-18T07:03:08.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:08.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:08.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:08.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:08.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:03:08.841+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:08.845119+0000) 2019-10-18T07:03:09.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:09.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:09.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:09.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:09.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:09.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:10.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:10.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:10.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:03:10.881+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:10.882581+0000) 2019-10-18T07:03:11.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:11.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:11.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:11.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:12.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:12.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:12.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:12.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:13.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:13.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:13.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:13.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:14.104 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:14.101+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:14.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:14.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:14.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:14.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:15.100 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:03:15.096+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:15.098570+0000) 2019-10-18T07:03:15.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:15.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:15.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:15.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:16.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:16.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:16.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:16.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:17.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:17.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:17.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:17.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:18.092 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:03:18.088+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:18.090642+0000) 2019-10-18T07:03:18.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:18.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:18.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:18.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:18.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:03:18.840+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:18.845226+0000) 2019-10-18T07:03:19.105 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:19.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:19.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:19.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:20.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:20.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:20.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:20.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:20.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:03:20.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:20.882686+0000) 2019-10-18T07:03:21.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:21.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:21.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:21.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:22.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:22.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:22.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:22.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:23.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:23.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:23.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:23.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:24.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:24.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:24.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:24.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:24.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:24.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:25.100 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:03:25.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:25.098781+0000) 2019-10-18T07:03:25.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:25.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:25.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:25.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:26.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:26.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:26.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:26.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:27.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:27.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:27.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:27.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:28.092 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:03:28.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:28.090851+0000) 2019-10-18T07:03:28.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:28.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:28.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:28.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:28.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:03:28.840+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:28.845333+0000) 2019-10-18T07:03:29.106 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:29.100+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:29.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:29.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:29.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:29.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:30.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:30.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:30.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:30.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:30.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:03:30.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:30.882785+0000) 2019-10-18T07:03:31.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:31.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:31.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:31.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:32.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:32.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:32.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:32.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:33.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:33.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:33.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:33.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:34.108 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:34.104+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:34.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:34.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:34.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:34.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:35.100 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:03:35.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:35.098964+0000) 2019-10-18T07:03:35.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:35.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:35.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:35.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:36.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:36.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:36.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:36.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:37.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:37.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:37.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:37.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:38.092 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:03:38.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:38.091065+0000) 2019-10-18T07:03:38.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:38.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:38.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:38.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:38.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:03:38.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:38.845437+0000) 2019-10-18T07:03:39.109 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:39.104+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:39.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:39.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:39.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:39.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:40.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:40.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:40.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:40.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:40.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:03:40.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:40.882891+0000) 2019-10-18T07:03:41.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:41.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:41.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:41.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:42.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:42.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:42.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:42.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:43.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:43.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:43.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:43.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:44.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:44.104+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:44.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:44.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:44.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:44.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:45.100 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:03:45.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:45.099149+0000) 2019-10-18T07:03:45.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:45.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:45.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:45.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:46.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:46.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:46.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:46.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:47.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:47.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:47.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:47.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:48.093 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:03:48.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:48.091275+0000) 2019-10-18T07:03:48.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:48.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:48.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:48.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:48.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:03:48.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:48.845549+0000) 2019-10-18T07:03:49.110 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:49.104+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:49.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:49.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:49.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:49.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:50.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:50.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:50.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:50.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:50.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:03:50.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:50.882994+0000) 2019-10-18T07:03:51.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:51.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:51.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:51.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:52.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:52.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:52.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:52.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:53.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:53.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:53.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:53.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:54.111 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:54.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:54.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:54.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:54.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:54.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:55.101 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:03:55.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:55.099358+0000) 2019-10-18T07:03:55.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:55.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:55.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:55.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:56.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:56.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:56.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:56.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:57.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:57.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:57.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:57.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:58.094 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:03:58.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:58.091492+0000) 2019-10-18T07:03:58.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:58.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:58.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:58.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:03:58.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:03:58.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:03:58.845646+0000) 2019-10-18T07:03:59.112 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:03:59.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:03:59.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:59.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:03:59.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:03:59.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:00.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:00.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:00.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:00.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:00.884 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:04:00.879+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:00.883107+0000) 2019-10-18T07:04:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:01.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:01.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:01.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:02.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:02.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:02.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:02.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:03.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:03.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:03.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:03.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:04.113 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:04.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:04.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:04.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:04.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:04.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:05.101 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:05.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:05.099543+0000) 2019-10-18T07:04:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:05.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:05.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:06.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:06.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:06.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:06.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:07.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:07.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:07.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:07.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:08.093 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:04:08.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:08.091673+0000) 2019-10-18T07:04:08.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:08.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:08.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:08.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:08.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:04:08.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:08.845747+0000) 2019-10-18T07:04:09.114 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:09.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:09.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:09.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:09.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:09.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:10.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:10.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:10.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:10.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:10.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:04:10.879+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:10.883206+0000) 2019-10-18T07:04:11.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:11.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:11.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:11.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:12.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:12.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:12.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:12.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:13.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:13.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:13.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:13.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:14.115 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:14.107+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:14.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:14.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:14.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:14.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:15.101 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:15.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:15.099726+0000) 2019-10-18T07:04:15.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:15.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:15.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:15.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:16.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:16.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:16.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:16.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:17.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:17.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:17.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:17.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:18.093 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:04:18.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:18.091851+0000) 2019-10-18T07:04:18.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:18.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:18.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:18.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:18.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:04:18.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:18.845852+0000) 2019-10-18T07:04:19.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:19.111+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:19.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:19.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:19.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:19.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:20.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:20.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:20.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:20.479+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:20.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:04:20.879+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:20.883310+0000) 2019-10-18T07:04:21.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:21.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:21.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:21.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:22.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:22.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T06:53:47.418993+0000 (oldest deadline 2019-10-18T06:54:07.418993+0000) 2019-10-18T07:04:22.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:22.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:23.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:23.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:24.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:24.111+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:24.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:24.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:25.101 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:25.095+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:25.099910+0000) 2019-10-18T07:04:25.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:25.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:26.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:26.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:27.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:27.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:28.093 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:04:28.087+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:28.092036+0000) 2019-10-18T07:04:28.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:28.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:28.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:04:28.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:28.845955+0000) 2019-10-18T07:04:29.117 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:29.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:29.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:29.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:30.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:30.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:30.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:04:30.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:30.883413+0000) 2019-10-18T07:04:31.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:31.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:32.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:32.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:33.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:33.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:34.118 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:34.110+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:34.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:34.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:35.101 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:35.094+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:35.100106+0000) 2019-10-18T07:04:35.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:35.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:36.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:36.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:37.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:37.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:38.094 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:04:38.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:38.092217+0000) 2019-10-18T07:04:38.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:38.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:38.847 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:04:38.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:38.846061+0000) 2019-10-18T07:04:39.119 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:39.114+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:39.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:39.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:40.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:40.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:40.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:04:40.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:40.883514+0000) 2019-10-18T07:04:41.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:41.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:42.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:42.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:43.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:43.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:44.121 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:44.114+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:44.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:44.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:45.103 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:45.094+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:45.100284+0000) 2019-10-18T07:04:45.103 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:45.094+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:45.101467+0000) 2019-10-18T07:04:45.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:45.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:45.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:45.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:46.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:46.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:46.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:46.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:47.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:47.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:47.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:47.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:48.095 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:04:48.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:48.092429+0000) 2019-10-18T07:04:48.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:48.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:48.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:48.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:48.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:04:48.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:48.846167+0000) 2019-10-18T07:04:49.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:49.114+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:49.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:49.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:49.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:49.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:49.602 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:49.594+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:49.600699+0000) 2019-10-18T07:04:50.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:50.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:50.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:50.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:50.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:04:50.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:50.883616+0000) 2019-10-18T07:04:51.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:51.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:51.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:51.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:52.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:52.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:52.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:52.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:53.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:53.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:53.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:53.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:54.122 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:54.114+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:54.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:54.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:54.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:55.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:55.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:55.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:55.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:56.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:56.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:56.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:56.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:57.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:57.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:57.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:57.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:58.094 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:04:58.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:58.092614+0000) 2019-10-18T07:04:58.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:58.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:58.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:58.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:58.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:04:58.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:58.846271+0000) 2019-10-18T07:04:59.123 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:04:59.118+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:04:59.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:59.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:04:59.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:04:59.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:04:59.602 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:04:59.594+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:04:59.600893+0000) 2019-10-18T07:05:00.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:00.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:00.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:00.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:00.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:05:00.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:00.883715+0000) 2019-10-18T07:05:01.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:01.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:01.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:01.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:02.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:02.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:02.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:02.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:03.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:03.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:03.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:03.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:04.124 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:04.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:04.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:04.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:04.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:04.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:05.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:05.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:05.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:05.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:06.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:06.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:06.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:06.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:07.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:07.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:07.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:07.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:08.094 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:05:08.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:08.092798+0000) 2019-10-18T07:05:08.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:08.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:08.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:08.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:08.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:05:08.841+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:08.846379+0000) 2019-10-18T07:05:09.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:09.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:09.603 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:09.594+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:09.601101+0000) 2019-10-18T07:05:09.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:09.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:09.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:09.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:10.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:10.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:10.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:10.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:10.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:05:10.877+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:10.883824+0000) 2019-10-18T07:05:11.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:11.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:11.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:11.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:12.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:12.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:12.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:12.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:13.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:13.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:13.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:13.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:14.125 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:14.117+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:14.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:14.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:14.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:14.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:15.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:15.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:15.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:15.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:16.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:16.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:16.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:16.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:17.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:17.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:17.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:17.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:18.094 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:05:18.086+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:18.093014+0000) 2019-10-18T07:05:18.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:18.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:18.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:18.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:18.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:05:18.841+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:18.846488+0000) 2019-10-18T07:05:19.127 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:19.121+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:19.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:19.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:19.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:19.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:19.602 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:19.594+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:19.601274+0000) 2019-10-18T07:05:19.603 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:19.594+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:19.602130+0000) 2019-10-18T07:05:20.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:20.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:20.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:20.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:20.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:05:20.877+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:20.883929+0000) 2019-10-18T07:05:21.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:21.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:21.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:21.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:22.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:22.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:22.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:22.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:23.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:23.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:23.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:23.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:24.103 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:24.094+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:24.101715+0000) 2019-10-18T07:05:24.128 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:24.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:24.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:24.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:24.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:24.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:25.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:25.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:25.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:25.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:26.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:26.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:26.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:26.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:27.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:27.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:27.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:27.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:28.094 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:05:28.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:28.093199+0000) 2019-10-18T07:05:28.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:28.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:28.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:28.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:28.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:05:28.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:28.846592+0000) 2019-10-18T07:05:29.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:29.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:29.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:29.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:29.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:29.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:30.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:30.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:30.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:30.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:30.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:05:30.881+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:30.884035+0000) 2019-10-18T07:05:31.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:31.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:31.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:31.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:32.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:32.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:32.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:32.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:33.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:33.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:33.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:33.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:34.103 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:34.098+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:34.101895+0000) 2019-10-18T07:05:34.129 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:34.125+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:34.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:34.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:34.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:34.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:35.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:35.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:35.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:35.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:36.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:36.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:36.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:36.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:37.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:37.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:37.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:37.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:38.095 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:05:38.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:38.093387+0000) 2019-10-18T07:05:38.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:38.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:38.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:38.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:38.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:05:38.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:38.846699+0000) 2019-10-18T07:05:39.130 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:39.124+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:39.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:39.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:39.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:39.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:40.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:40.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:40.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:40.885 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:05:40.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:40.884141+0000) 2019-10-18T07:05:41.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:41.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:41.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:41.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:42.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:42.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:42.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:42.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:43.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:43.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:43.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:43.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:44.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:44.097+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:44.102100+0000) 2019-10-18T07:05:44.131 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:44.128+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:44.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:44.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:44.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:44.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:45.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:45.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:45.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:45.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:46.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:46.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:46.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:46.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:47.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:47.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:47.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:47.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:48.095 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:05:48.089+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:48.093598+0000) 2019-10-18T07:05:48.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:48.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:48.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:48.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:48.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:05:48.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:48.846807+0000) 2019-10-18T07:05:49.132 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:49.128+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:49.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:49.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:49.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:49.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:50.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:50.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:50.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:50.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:50.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:05:50.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:50.884255+0000) 2019-10-18T07:05:51.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:51.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:51.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:51.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:52.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:52.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:52.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:52.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:53.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:53.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:53.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:53.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:54.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:05:54.101+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:54.102252+0000) 2019-10-18T07:05:54.133 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:54.128+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:54.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:54.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:54.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:54.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:55.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:55.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:55.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:55.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:56.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:56.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:56.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:56.400+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:57.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:57.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:57.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:57.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:58.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:05:58.089+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:58.093790+0000) 2019-10-18T07:05:58.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:58.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:58.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:58.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:05:58.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:05:58.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:05:58.846913+0000) 2019-10-18T07:05:59.134 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:05:59.128+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:05:59.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:59.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:05:59.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:05:59.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:00.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:00.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:00.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:00.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:00.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:06:00.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:00.884362+0000) 2019-10-18T07:06:01.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:01.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:01.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:01.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:02.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:02.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:02.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:02.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:03.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:03.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:03.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:03.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:04.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:06:04.101+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:04.102433+0000) 2019-10-18T07:06:04.135 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:04.128+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:04.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:04.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:04.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:04.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:05.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:05.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:05.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:05.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:06.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:06.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:06.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:06.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:07.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:07.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:07.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:07.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:08.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:06:08.089+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:08.093977+0000) 2019-10-18T07:06:08.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:08.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:08.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:08.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:08.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:06:08.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:08.847017+0000) 2019-10-18T07:06:09.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:09.132+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:09.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:09.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:09.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:09.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:10.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:10.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:10.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:10.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 540 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:10.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:06:10.880+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:10.884470+0000) 2019-10-18T07:06:11.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:11.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:11.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:11.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:12.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:12.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:12.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:12.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:13.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:13.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:13.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:13.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:14.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:06:14.101+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:14.102619+0000) 2019-10-18T07:06:14.136 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:14.131+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:14.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:14.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:14.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:14.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:15.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:15.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:15.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:15.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:16.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:16.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:16.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:16.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:17.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:17.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:18.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:06:18.093+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:18.094223+0000) 2019-10-18T07:06:18.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:18.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:18.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:18.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:18.848 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:06:18.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:18.847123+0000) 2019-10-18T07:06:19.137 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:19.131+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:19.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:19.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:19.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:19.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:20.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:20.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:20.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:20.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:20.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:06:20.879+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:20.884577+0000) 2019-10-18T07:06:21.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:21.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:21.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:21.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:22.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:22.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:22.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:22.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:23.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:23.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:23.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:23.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:24.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:06:24.101+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:24.102813+0000) 2019-10-18T07:06:24.138 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:24.131+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:24.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:24.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:24.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:24.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:25.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:25.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:25.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:25.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:26.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:26.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:26.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:26.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:27.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:27.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:27.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:27.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:28.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:06:28.093+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:28.094400+0000) 2019-10-18T07:06:28.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:28.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:28.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:28.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:28.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:06:28.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:28.847225+0000) 2019-10-18T07:06:29.139 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:29.135+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:29.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:29.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:29.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:29.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:30.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:30.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:30.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:30.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:30.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:06:30.879+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:30.884681+0000) 2019-10-18T07:06:31.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:31.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:31.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:31.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:32.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:32.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:32.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:32.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:33.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:33.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:33.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:33.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:34.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:06:34.101+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:34.103024+0000) 2019-10-18T07:06:34.140 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:34.135+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:34.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:34.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:34.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:34.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:35.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:35.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:35.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:35.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:36.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:36.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:36.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:36.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:37.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:37.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:37.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:37.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:38.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:06:38.093+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:38.094576+0000) 2019-10-18T07:06:38.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:38.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:38.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:38.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:38.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:06:38.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:38.847328+0000) 2019-10-18T07:06:39.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:39.135+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:39.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:39.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:39.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:39.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:40.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:40.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:40.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:40.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:40.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:06:40.879+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:40.884780+0000) 2019-10-18T07:06:41.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:41.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:41.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:41.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:42.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:42.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:42.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:42.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:43.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:43.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:43.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:43.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:44.104 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:06:44.101+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:44.103209+0000) 2019-10-18T07:06:44.141 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:44.135+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:44.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:44.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:44.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:44.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:45.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:45.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:45.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:45.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:46.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:46.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:46.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:46.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:47.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:47.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:47.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:47.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:48.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:06:48.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:48.094738+0000) 2019-10-18T07:06:48.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:48.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:48.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:48.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:48.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:06:48.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:48.847437+0000) 2019-10-18T07:06:49.142 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:49.134+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:49.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:49.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:49.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:49.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:50.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:50.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:50.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:50.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:50.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:06:50.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:50.884885+0000) 2019-10-18T07:06:51.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:51.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:51.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:51.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:52.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:52.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:52.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:52.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:53.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:53.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:53.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:53.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:54.105 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:06:54.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:54.103399+0000) 2019-10-18T07:06:54.144 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:54.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:54.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:54.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:54.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:54.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:55.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:55.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:55.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:55.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:56.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:56.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:56.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:56.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:57.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:57.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:57.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:57.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:58.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:06:58.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:58.094921+0000) 2019-10-18T07:06:58.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:58.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:58.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:58.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:06:58.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:06:58.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:06:58.847543+0000) 2019-10-18T07:06:59.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:06:59.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:06:59.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:59.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:06:59.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:06:59.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:00.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:00.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:00.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:00.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:00.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:07:00.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:00.884982+0000) 2019-10-18T07:07:01.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:01.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:01.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:01.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:02.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:02.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:02.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:02.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:03.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:03.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:03.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:03.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:04.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:07:04.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:04.103606+0000) 2019-10-18T07:07:04.145 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:04.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:04.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:04.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:04.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:04.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:05.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:05.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:05.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:05.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:06.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:06.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:06.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:06.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:07.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:07.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:07.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:07.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:08.096 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:07:08.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:08.095101+0000) 2019-10-18T07:07:08.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:08.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:08.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:08.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:08.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:07:08.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:08.847653+0000) 2019-10-18T07:07:09.146 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:09.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:09.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:09.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:09.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:09.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:10.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:10.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:10.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:10.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:10.886 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:07:10.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:10.885085+0000) 2019-10-18T07:07:11.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:11.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:11.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:11.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:12.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:12.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:12.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:12.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:13.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:13.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:13.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:13.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:14.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:07:14.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:14.103779+0000) 2019-10-18T07:07:14.147 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:14.138+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:14.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:14.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:14.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:14.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:15.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:15.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:15.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:15.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:16.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:16.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:16.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:16.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:17.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:17.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:17.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:17.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:18.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:07:18.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:18.095286+0000) 2019-10-18T07:07:18.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:18.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:18.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:18.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:18.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:07:18.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:18.847757+0000) 2019-10-18T07:07:19.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:19.142+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:19.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:19.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:19.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:19.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:20.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:20.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:20.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:20.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:20.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:07:20.878+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:20.885187+0000) 2019-10-18T07:07:21.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:21.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:21.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:21.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:22.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:22.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:22.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:22.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:23.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:23.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:23.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:23.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:24.105 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:07:24.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:24.103978+0000) 2019-10-18T07:07:24.148 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:24.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:24.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:24.289+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:24.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:24.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:25.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:25.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:25.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:25.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:26.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:26.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:26.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:26.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:27.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:27.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:27.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:27.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:28.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:07:28.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:28.095468+0000) 2019-10-18T07:07:28.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:28.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:28.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:28.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:28.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:07:28.841+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:28.847861+0000) 2019-10-18T07:07:29.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:29.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:29.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:29.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:29.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:29.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:30.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:30.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:30.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:30.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:30.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:07:30.877+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:30.885289+0000) 2019-10-18T07:07:31.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:31.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:31.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:31.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:32.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:32.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:32.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:32.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:33.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:33.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:33.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:33.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:34.105 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:07:34.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:34.104125+0000) 2019-10-18T07:07:34.150 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:34.141+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:34.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:34.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:34.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:34.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:35.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:35.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:35.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:35.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:36.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:36.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:36.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:36.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:37.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:37.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:37.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:37.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:38.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:07:38.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:38.095612+0000) 2019-10-18T07:07:38.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:38.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:38.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:38.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:38.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:07:38.841+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:38.847972+0000) 2019-10-18T07:07:39.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:39.145+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:39.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:39.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:39.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:39.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:40.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:40.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:40.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:40.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:40.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:07:40.877+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:40.885384+0000) 2019-10-18T07:07:41.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:41.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:41.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:41.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:42.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:42.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:42.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:42.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:43.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:43.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:43.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:43.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:44.106 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:07:44.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:44.104318+0000) 2019-10-18T07:07:44.152 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:44.149+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:44.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:44.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:44.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:44.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:45.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:45.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:45.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:45.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:46.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:46.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:46.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:46.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:47.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:47.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:47.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:47.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:48.097 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:07:48.092+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:48.095793+0000) 2019-10-18T07:07:48.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:48.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:48.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:48.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:48.849 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:07:48.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:48.848070+0000) 2019-10-18T07:07:49.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:49.149+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:49.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:49.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:49.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:49.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:50.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:50.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:50.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:50.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:50.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:07:50.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:50.885487+0000) 2019-10-18T07:07:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:51.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:51.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:51.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:52.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:52.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:52.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:52.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:53.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:53.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:53.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:53.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:54.106 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:07:54.100+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:54.104511+0000) 2019-10-18T07:07:54.153 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:54.149+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:54.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:54.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:54.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:54.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:55.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:55.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:55.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:55.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:56.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:56.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:56.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:56.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:57.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:57.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:57.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:57.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:58.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:07:58.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:58.096008+0000) 2019-10-18T07:07:58.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:58.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:58.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:58.501+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:07:58.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:07:58.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:07:58.848188+0000) 2019-10-18T07:07:59.154 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:07:59.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:07:59.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:59.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:07:59.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:07:59.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:00.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:00.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:00.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:00.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:00.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:08:00.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:00.885589+0000) 2019-10-18T07:08:01.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:01.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:01.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:01.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:02.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:02.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:02.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:02.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:03.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:03.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:03.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:03.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:04.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:08:04.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:04.104669+0000) 2019-10-18T07:08:04.155 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:04.148+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:04.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:04.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:04.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:05.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:05.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:05.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:05.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:06.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:06.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:06.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:06.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:07.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:07.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:07.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:07.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:08.098 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:08:08.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:08.096191+0000) 2019-10-18T07:08:08.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:08.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:08.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:08.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:08.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:08:08.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:08.848298+0000) 2019-10-18T07:08:09.156 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:09.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:09.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:09.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:09.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:09.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:10.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:10.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:10.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:10.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:10.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:08:10.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:10.885694+0000) 2019-10-18T07:08:11.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:11.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:11.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:11.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:12.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:12.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:12.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:12.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:13.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:13.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:13.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:13.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:14.106 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:08:14.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:14.104850+0000) 2019-10-18T07:08:14.157 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:14.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:14.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:14.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:14.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:14.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:15.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:15.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:15.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:15.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:16.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:16.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:16.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:16.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:17.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:17.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:17.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:17.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:18.098 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:08:18.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:18.096400+0000) 2019-10-18T07:08:18.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:18.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:18.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:18.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:18.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:08:18.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:18.848407+0000) 2019-10-18T07:08:19.158 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:19.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:19.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:19.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:19.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:19.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:20.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:20.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:20.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:20.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:20.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:08:20.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:20.885792+0000) 2019-10-18T07:08:21.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:21.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:21.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:21.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:22.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:22.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:22.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:22.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:23.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:23.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:23.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:23.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:24.106 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:08:24.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:24.105031+0000) 2019-10-18T07:08:24.158 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:24.152+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:24.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:24.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:24.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:25.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:25.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:25.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:25.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:26.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:26.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:26.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:26.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:27.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:27.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:27.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:27.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:28.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:08:28.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:28.096580+0000) 2019-10-18T07:08:28.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:28.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:28.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:28.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:28.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:08:28.844+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:28.848514+0000) 2019-10-18T07:08:29.159 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:29.156+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:29.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:29.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:29.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:29.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:30.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:30.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:30.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:30.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:30.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:08:30.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:30.885899+0000) 2019-10-18T07:08:31.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:31.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:31.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:31.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:32.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:32.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:32.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:32.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:33.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:33.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:33.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:33.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:34.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:08:34.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:34.105213+0000) 2019-10-18T07:08:34.160 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:34.155+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:34.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:34.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:34.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:34.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:35.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:35.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:35.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:35.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:36.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:36.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:36.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:36.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:37.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:37.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:37.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:37.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:38.098 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:08:38.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:38.096757+0000) 2019-10-18T07:08:38.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:08:38.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:38.848615+0000) 2019-10-18T07:08:38.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:38.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:38.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:38.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:39.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:39.155+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:39.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:39.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:39.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:39.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:40.887 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:08:40.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:40.886009+0000) 2019-10-18T07:08:40.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:40.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:40.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:40.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:41.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:41.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:41.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:41.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:42.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:42.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:42.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:42.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:44.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:44.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:44.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:44.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:44.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:08:44.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:44.105396+0000) 2019-10-18T07:08:44.161 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:44.155+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:45.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:45.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:45.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:45.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:46.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:46.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:46.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:46.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:47.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:47.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:47.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:47.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:47.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:47.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:47.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:47.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:48.098 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:08:48.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:48.096934+0000) 2019-10-18T07:08:48.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:08:48.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:48.848723+0000) 2019-10-18T07:08:49.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:49.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:49.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:49.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:49.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:49.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:50.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:50.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:50.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:50.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:50.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:08:50.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:50.886117+0000) 2019-10-18T07:08:51.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:51.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:51.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:51.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:52.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:52.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:52.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:52.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:53.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:53.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:53.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:53.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:54.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:08:54.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:54.105583+0000) 2019-10-18T07:08:54.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:54.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:54.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:54.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:54.163 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:54.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:08:55.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:55.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:55.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:55.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:56.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:56.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:56.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:56.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:57.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:57.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:57.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:57.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:58.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:58.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:58.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:58.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:58.098 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:08:58.091+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:58.097117+0000) 2019-10-18T07:08:58.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:08:58.843+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:08:58.848830+0000) 2019-10-18T07:08:59.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:59.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:08:59.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:08:59.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:08:59.165 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:08:59.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:00.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:00.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:00.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:00.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:00.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:09:00.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:00.886224+0000) 2019-10-18T07:09:01.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:01.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:01.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:01.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:02.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:02.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:02.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:02.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:03.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:03.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:03.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:03.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:04.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:04.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:04.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:04.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:04.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:09:04.099+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:04.105779+0000) 2019-10-18T07:09:04.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:04.159+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:05.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:05.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:05.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:05.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:06.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:06.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:06.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:06.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:07.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:07.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:07.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:07.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:08.099 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:09:08.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:08.097298+0000) 2019-10-18T07:09:08.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:08.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:08.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:08.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:08.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:09:08.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:08.848933+0000) 2019-10-18T07:09:09.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:09.158+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:09.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:09.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:09.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:09.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:10.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:10.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:10.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:10.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:10.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:09:10.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:10.886322+0000) 2019-10-18T07:09:11.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:11.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:11.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:11.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:12.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:12.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:12.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:12.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:13.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:13.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:13.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:13.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:14.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:09:14.098+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:14.105923+0000) 2019-10-18T07:09:14.166 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:14.158+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:14.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:14.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:15.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:15.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:15.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:15.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:16.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:16.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:16.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:16.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:17.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:17.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:17.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:17.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:18.099 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:09:18.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:18.097478+0000) 2019-10-18T07:09:18.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:18.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:18.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:18.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:18.850 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:09:18.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:18.849031+0000) 2019-10-18T07:09:19.168 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:19.162+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:19.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:19.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:19.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:19.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:20.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:20.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:20.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:20.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:20.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:09:20.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:20.886423+0000) 2019-10-18T07:09:21.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:21.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:21.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:21.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:22.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:22.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:22.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:22.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:23.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:23.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:23.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:23.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:24.107 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:09:24.098+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:24.106063+0000) 2019-10-18T07:09:24.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:24.162+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:24.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:24.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:24.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:24.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:25.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:25.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:25.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:25.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:26.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:26.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:26.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:26.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:27.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:27.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:27.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:27.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:28.099 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:09:28.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:28.097661+0000) 2019-10-18T07:09:28.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:28.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:28.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:28.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:28.851 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:09:28.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:28.849135+0000) 2019-10-18T07:09:29.169 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:29.162+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:29.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:29.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:29.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:29.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:30.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:30.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:30.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:30.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:30.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:09:30.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:30.886527+0000) 2019-10-18T07:09:31.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:31.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:31.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:31.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:32.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:32.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:32.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:32.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:33.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:33.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:33.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:33.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:34.108 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:09:34.102+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:34.106192+0000) 2019-10-18T07:09:34.170 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:34.162+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:34.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:34.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:34.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:34.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:35.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:35.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:35.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:35.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:36.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:36.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:36.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:36.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:37.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:37.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:37.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:37.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:38.099 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:09:38.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:38.097849+0000) 2019-10-18T07:09:38.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:38.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:38.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:38.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:38.851 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:09:38.842+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:38.849240+0000) 2019-10-18T07:09:39.171 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:39.166+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:39.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:39.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:39.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:39.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:40.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:40.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:40.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:40.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:09:40.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:40.886625+0000) 2019-10-18T07:09:41.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:41.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:41.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:41.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:42.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:42.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:42.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:42.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:43.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:43.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:43.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:43.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:44.108 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:09:44.102+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:44.106314+0000) 2019-10-18T07:09:44.172 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:44.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:44.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:44.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:44.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:44.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:45.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:45.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:45.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:45.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:46.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:46.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:46.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:46.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:47.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:47.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:47.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:47.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:48.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:09:48.090+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:48.098031+0000) 2019-10-18T07:09:48.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:48.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:48.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:48.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:48.851 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:09:48.841+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:48.849349+0000) 2019-10-18T07:09:49.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:49.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:49.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:49.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:49.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:49.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:50.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:50.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:50.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:50.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:50.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:09:50.881+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:50.886730+0000) 2019-10-18T07:09:51.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:51.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:51.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:51.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:52.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:52.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:52.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:52.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:53.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:53.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:53.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:53.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:54.108 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:09:54.102+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:54.106439+0000) 2019-10-18T07:09:54.173 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:54.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:54.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:54.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:54.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:54.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:55.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:55.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:55.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:55.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:56.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:56.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:56.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:56.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:57.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:57.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:57.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:57.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:58.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:09:58.094+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:58.098242+0000) 2019-10-18T07:09:58.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:58.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:58.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:58.229+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:09:58.851 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:09:58.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:09:58.849470+0000) 2019-10-18T07:09:59.175 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:09:59.165+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:09:59.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:59.273+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:09:59.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:09:59.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:00.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:00.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:00.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:00.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:00.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:10:00.881+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:00.886837+0000) 2019-10-18T07:10:01.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:01.301+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:01.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:01.301+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:02.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:02.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:02.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:02.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:03.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:03.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:03.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:03.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:04.108 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:10:04.102+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:04.106578+0000) 2019-10-18T07:10:04.176 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:04.173+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:04.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:04.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:04.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:04.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:05.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:05.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:05.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:05.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:06.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:06.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:06.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:06.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:07.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:07.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:07.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:07.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:08.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:10:08.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:08.098426+0000) 2019-10-18T07:10:08.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:08.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:08.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:08.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:08.851 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:10:08.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:08.849580+0000) 2019-10-18T07:10:09.176 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:09.173+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:09.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:09.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:09.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:09.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:10.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:10.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:10.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:10.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:10.888 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:10:10.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:10.886938+0000) 2019-10-18T07:10:11.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:11.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:11.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:11.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:12.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:12.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:12.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:12.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:13.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:13.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:13.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:13.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:14.108 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:10:14.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:14.106775+0000) 2019-10-18T07:10:14.177 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:14.173+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:14.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:14.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:14.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:14.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:15.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:15.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:15.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:15.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:16.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:16.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:16.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:16.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:17.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:17.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:17.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:17.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:18.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:10:18.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:18.098609+0000) 2019-10-18T07:10:18.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:18.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:18.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:18.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:18.851 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:10:18.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:18.849684+0000) 2019-10-18T07:10:19.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:19.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:19.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:19.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:19.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:19.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:20.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:20.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:20.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:20.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:20.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:10:20.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:20.887041+0000) 2019-10-18T07:10:21.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:21.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:21.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:21.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:22.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:22.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:22.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:22.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:23.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:23.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:23.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:23.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:24.108 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:10:24.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:24.106959+0000) 2019-10-18T07:10:24.178 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:24.172+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:24.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:24.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:24.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:24.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:25.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:25.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:25.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:25.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:26.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:26.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:26.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:26.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:27.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:27.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:27.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:27.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:28.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:10:28.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:28.098817+0000) 2019-10-18T07:10:28.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:28.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:28.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:28.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:28.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:10:28.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:28.849788+0000) 2019-10-18T07:10:29.180 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:29.176+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:29.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:29.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:29.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:29.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:30.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:30.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:30.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:30.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:30.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:10:30.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:30.887140+0000) 2019-10-18T07:10:31.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:31.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:31.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:31.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:32.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:32.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:32.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:32.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:33.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:33.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:33.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:33.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:34.109 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:10:34.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:34.107166+0000) 2019-10-18T07:10:34.181 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:34.176+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:34.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:34.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:34.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:34.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:35.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:35.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:35.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:35.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:36.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:36.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:36.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:36.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:37.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:37.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:37.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:37.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:38.100 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:10:38.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:38.099001+0000) 2019-10-18T07:10:38.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:38.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:38.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:38.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:38.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:10:38.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:38.849895+0000) 2019-10-18T07:10:39.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:39.176+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:39.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:39.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:39.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:39.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:40.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:40.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:40.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:40.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:40.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:10:40.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:40.887243+0000) 2019-10-18T07:10:41.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:41.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:41.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:41.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:42.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:42.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:42.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:42.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:43.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:43.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:43.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:43.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:44.109 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:10:44.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:44.107337+0000) 2019-10-18T07:10:44.182 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:44.176+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:44.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:44.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:44.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:44.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:45.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:45.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:45.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:45.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:46.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:46.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:46.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:46.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:47.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:47.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:47.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:47.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:48.101 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:10:48.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:48.099180+0000) 2019-10-18T07:10:48.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:48.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:48.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:48.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:48.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:10:48.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:48.850005+0000) 2019-10-18T07:10:49.184 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:49.176+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:49.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:49.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:49.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:49.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:50.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:50.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:50.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:50.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:50.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:10:50.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:50.887346+0000) 2019-10-18T07:10:51.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:51.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:51.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:51.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:52.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:52.544+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:52.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:52.544+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:53.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:53.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:53.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:53.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:54.109 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:10:54.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:54.107472+0000) 2019-10-18T07:10:54.183 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:54.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:54.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:54.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:54.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:54.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:55.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:55.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:55.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:55.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:56.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:56.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:56.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:56.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:57.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:57.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:57.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:57.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:58.101 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:10:58.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:58.099358+0000) 2019-10-18T07:10:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:58.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:58.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:58.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:10:58.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:10:58.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:10:58.850107+0000) 2019-10-18T07:10:59.185 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:10:59.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:10:59.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:59.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:10:59.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:10:59.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:00.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:00.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:00.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:00.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:00.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:11:00.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:00.887453+0000) 2019-10-18T07:11:01.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:01.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:01.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:01.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:02.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:02.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:02.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:02.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:03.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:03.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:03.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:03.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:04.109 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:11:04.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:04.107649+0000) 2019-10-18T07:11:04.186 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:04.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:04.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:04.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:04.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:04.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:05.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:05.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:05.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:05.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:06.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:06.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:06.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:06.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:07.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:07.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:07.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:07.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:08.101 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:11:08.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:08.099537+0000) 2019-10-18T07:11:08.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:08.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:08.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:08.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:08.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:11:08.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:08.850212+0000) 2019-10-18T07:11:09.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:09.179+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:09.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:09.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:09.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:09.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:10.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:10.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:10.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:10.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:10.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:11:10.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:10.887559+0000) 2019-10-18T07:11:11.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:11.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:11.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:11.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:12.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:12.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:12.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:12.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:13.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:13.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:13.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:13.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:14.109 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:11:14.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:14.107795+0000) 2019-10-18T07:11:14.187 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:14.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:14.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:14.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:14.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:14.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:15.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:15.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:15.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:15.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:16.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:16.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:16.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:16.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:17.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:17.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:17.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:17.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:18.101 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:11:18.097+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:18.099751+0000) 2019-10-18T07:11:18.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:18.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:18.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:18.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:18.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:11:18.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:18.850319+0000) 2019-10-18T07:11:19.188 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:19.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:19.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:19.635+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:19.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:19.635+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:20.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:20.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:20.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:20.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:20.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:11:20.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:20.887665+0000) 2019-10-18T07:11:21.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:21.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:21.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:21.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:22.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:22.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:22.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:22.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:23.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:23.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:23.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:23.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:24.109 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:11:24.105+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:24.107960+0000) 2019-10-18T07:11:24.189 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:24.183+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:24.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:24.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:24.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:24.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:25.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:25.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:25.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:25.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:26.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:26.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:26.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:26.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:27.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:27.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:27.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:27.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:28.101 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:11:28.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:28.099930+0000) 2019-10-18T07:11:28.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:28.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:28.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:28.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:28.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:11:28.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:28.850425+0000) 2019-10-18T07:11:29.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:29.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:29.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:29.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:29.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:29.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:30.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:30.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:30.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:30.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:30.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:11:30.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:30.887772+0000) 2019-10-18T07:11:31.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:31.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:31.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:31.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:32.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:32.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:32.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:32.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:33.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:33.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:33.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:33.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:34.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:11:34.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:34.108131+0000) 2019-10-18T07:11:34.190 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:34.182+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:34.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:34.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:34.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:34.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:35.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:35.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:35.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:35.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:36.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:36.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:36.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:36.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:37.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:37.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:37.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:37.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:38.101 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:11:38.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:38.100114+0000) 2019-10-18T07:11:38.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:38.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:38.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:38.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:38.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:11:38.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:38.850532+0000) 2019-10-18T07:11:39.191 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:39.186+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:39.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:39.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:39.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:39.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:40.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:40.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:40.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:40.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:40.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:11:40.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:40.887875+0000) 2019-10-18T07:11:41.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:41.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:41.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:41.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:42.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:42.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:42.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:42.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:43.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:43.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:43.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:43.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:44.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:11:44.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:44.108287+0000) 2019-10-18T07:11:44.192 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:44.186+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:44.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:44.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:44.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:44.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:45.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:45.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:45.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:45.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:46.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:46.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:46.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:46.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:47.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:47.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:47.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:47.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:48.102 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:11:48.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:48.100293+0000) 2019-10-18T07:11:48.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:48.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:48.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:48.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:48.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:11:48.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:48.850643+0000) 2019-10-18T07:11:49.193 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:49.186+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:49.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:49.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:49.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:49.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:50.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:50.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:50.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:50.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:50.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:11:50.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:50.887977+0000) 2019-10-18T07:11:51.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:51.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:51.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:51.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:52.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:52.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:52.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:52.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:53.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:53.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:53.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:53.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:54.110 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:11:54.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:54.108444+0000) 2019-10-18T07:11:54.194 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:54.186+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:54.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:54.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:54.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:54.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:55.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:55.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:55.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:55.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:56.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:56.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:56.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:56.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:57.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:57.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:57.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:57.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:58.102 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:11:58.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:58.100473+0000) 2019-10-18T07:11:58.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:58.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:58.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:58.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:11:58.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:11:58.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:11:58.850744+0000) 2019-10-18T07:11:59.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:11:59.186+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:11:59.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:59.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:11:59.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:11:59.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:00.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:00.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:00.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:00.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:00.889 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:12:00.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:00.888082+0000) 2019-10-18T07:12:01.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:01.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:01.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:01.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:02.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:02.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:02.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:02.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:03.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:03.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:03.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:03.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:04.110 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:12:04.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:04.108586+0000) 2019-10-18T07:12:04.195 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:04.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:04.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:04.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:04.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:04.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:05.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:05.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:05.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:05.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:06.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:06.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:06.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:06.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:07.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:07.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:07.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:07.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:08.102 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:12:08.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:08.100656+0000) 2019-10-18T07:12:08.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:08.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:08.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:08.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:08.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:12:08.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:08.850850+0000) 2019-10-18T07:12:09.196 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:09.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:09.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:09.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:09.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:09.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:10.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:10.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:10.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:10.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:10.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:12:10.881+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:10.888199+0000) 2019-10-18T07:12:11.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:11.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:11.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:11.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:12.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:12.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:12.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:12.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:13.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:13.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:13.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:13.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:14.110 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:12:14.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:14.108755+0000) 2019-10-18T07:12:14.197 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:14.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:14.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:14.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:14.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:14.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:15.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:15.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:15.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:15.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:16.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:16.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:16.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:16.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:17.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:17.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:17.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:17.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:18.102 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:12:18.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:18.100835+0000) 2019-10-18T07:12:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:18.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:18.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:18.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:12:18.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:18.850956+0000) 2019-10-18T07:12:19.198 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:19.189+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:19.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:19.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:19.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:19.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:20.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:20.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:20.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:20.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:20.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:12:20.881+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:20.888299+0000) 2019-10-18T07:12:21.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:21.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:21.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:21.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:22.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:22.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:22.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:22.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:23.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:23.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:23.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:23.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:24.110 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:12:24.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:24.108941+0000) 2019-10-18T07:12:24.199 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:24.197+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:24.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:24.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:24.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:24.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:25.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:25.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:25.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:25.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:26.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:26.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:26.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:26.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:27.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:27.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:27.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:27.869+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:28.103 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:12:28.096+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:28.101017+0000) 2019-10-18T07:12:28.852 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:12:28.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:28.851067+0000) 2019-10-18T07:12:28.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:28.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:28.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:28.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:29.200 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:29.197+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:29.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:29.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:29.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:29.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:30.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:12:30.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:30.888404+0000) 2019-10-18T07:12:30.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:30.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:30.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:30.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:31.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:31.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:31.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:31.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:32.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:32.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:32.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:32.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:33.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:33.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:33.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:33.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:34.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:12:34.104+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:34.109089+0000) 2019-10-18T07:12:34.201 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:34.197+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:34.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:34.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:34.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:34.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:35.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:35.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:35.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:35.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:36.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:36.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:36.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:36.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:37.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:37.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:37.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:37.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:38.103 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:12:38.095+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:38.101197+0000) 2019-10-18T07:12:38.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:38.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:38.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:38.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:38.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:12:38.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:38.851175+0000) 2019-10-18T07:12:39.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:39.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:39.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:39.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:39.848+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:40.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:40.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:40.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:40.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:40.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:12:40.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:40.888502+0000) 2019-10-18T07:12:41.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:41.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:41.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:41.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:42.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:42.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:42.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:42.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:43.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:43.908+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:43.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:43.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:44.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:12:44.103+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:44.109227+0000) 2019-10-18T07:12:44.202 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:44.196+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:44.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:44.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:44.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:44.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:45.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:45.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:45.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:45.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:46.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:46.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:46.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:46.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:47.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:47.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:48.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:47.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:48.103 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:12:48.095+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:48.101377+0000) 2019-10-18T07:12:48.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:12:48.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:48.851282+0000) 2019-10-18T07:12:48.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:48.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:48.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:48.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:49.203 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:49.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:49.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:49.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:49.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:49.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:50.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:12:50.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:50.888607+0000) 2019-10-18T07:12:50.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:50.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:50.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:50.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:51.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:51.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:51.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:51.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:52.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:52.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:52.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:52.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:53.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:53.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:53.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:53.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:54.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:12:54.103+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:54.109363+0000) 2019-10-18T07:12:54.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:54.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:12:54.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:54.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:54.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:54.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:56.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:56.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:56.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:56.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:57.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:57.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:57.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:57.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:58.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:58.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:58.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:58.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:58.103 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:12:58.095+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:58.101564+0000) 2019-10-18T07:12:58.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:12:58.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:12:58.851388+0000) 2019-10-18T07:12:59.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:59.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:12:59.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:12:59.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:12:59.205 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:12:59.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:00.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:00.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:00.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:00.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:00.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:13:00.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:00.888708+0000) 2019-10-18T07:13:01.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:01.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:01.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:01.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:02.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:02.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:02.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:02.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:03.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:03.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:03.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:03.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:04.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:04.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:04.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:04.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:04.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:13:04.103+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:04.109501+0000) 2019-10-18T07:13:04.206 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:04.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:05.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:05.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:05.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:05.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:06.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:06.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:06.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:06.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:07.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:07.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:07.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:07.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:08.103 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:13:08.095+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:08.101739+0000) 2019-10-18T07:13:08.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:08.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:08.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:08.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:08.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:13:08.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:08.851498+0000) 2019-10-18T07:13:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:09.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:09.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:09.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:09.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:09.200+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:10.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:10.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:10.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:10.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:10.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:13:10.884+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:10.888809+0000) 2019-10-18T07:13:11.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:11.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:11.185 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:11.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:12.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:12.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:12.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:12.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:13.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:13.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:13.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:13.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:14.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:13:14.103+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:14.109672+0000) 2019-10-18T07:13:14.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:14.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:14.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:14.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:14.207 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:14.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:15.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:15.147+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:15.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:15.147+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:16.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:16.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:16.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:16.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:17.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:17.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:17.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:17.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:18.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:18.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:18.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:18.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:18.103 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:13:18.095+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:18.101941+0000) 2019-10-18T07:13:18.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:13:18.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:18.851605+0000) 2019-10-18T07:13:19.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:19.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:19.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:19.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:19.208 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:19.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:20.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:20.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:20.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:20.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:13:20.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:20.888911+0000) 2019-10-18T07:13:21.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:21.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:21.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:21.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:22.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:22.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:22.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:22.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:23.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:23.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:23.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:23.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:24.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:24.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:24.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:24.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:24.111 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:13:24.103+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:24.109842+0000) 2019-10-18T07:13:24.209 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:24.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:25.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:25.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:25.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:25.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:26.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:26.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:26.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:26.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:26.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:26.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:26.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:26.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:27.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:27.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:27.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:27.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:28.104 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:13:28.095+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:28.102126+0000) 2019-10-18T07:13:28.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:13:28.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:28.851705+0000) 2019-10-18T07:13:28.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:28.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:28.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:28.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:29.210 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:29.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:29.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:29.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:29.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:29.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:30.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:13:30.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:30.889017+0000) 2019-10-18T07:13:30.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:30.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:30.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:30.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:31.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:31.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:31.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:31.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:32.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:32.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:32.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:32.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:33.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:33.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:33.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:33.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:34.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:13:34.103+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:34.110008+0000) 2019-10-18T07:13:34.211 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:34.203+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:34.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:34.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:34.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:34.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:35.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:35.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:35.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:35.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:36.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:36.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:36.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:36.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:38.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:38.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:38.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:38.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:38.104 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:13:38.099+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:38.102356+0000) 2019-10-18T07:13:38.853 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:13:38.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:38.851803+0000) 2019-10-18T07:13:39.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:38.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:39.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:38.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:39.213 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:39.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:40.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:40.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:40.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:40.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:40.890 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:13:40.883+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:40.889118+0000) 2019-10-18T07:13:41.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:41.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:41.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:41.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:42.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:42.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:42.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:42.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:43.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:43.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:43.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:43.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:44.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:44.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:44.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:44.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:44.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:13:44.107+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:44.110224+0000) 2019-10-18T07:13:44.212 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:44.207+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:45.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:45.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:45.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:45.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:46.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:46.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:46.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:46.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:47.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:47.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:47.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:47.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:48.104 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:13:48.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:48.102551+0000) 2019-10-18T07:13:48.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:48.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:48.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:48.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:48.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:13:48.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:48.851912+0000) 2019-10-18T07:13:49.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:49.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:49.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:49.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:49.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:49.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:50.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:50.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:50.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:50.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:50.898 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:13:50.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:50.889221+0000) 2019-10-18T07:13:51.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:51.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:51.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:51.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:52.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:52.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:52.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:52.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:53.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:53.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:53.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:53.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:54.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:13:54.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:54.110390+0000) 2019-10-18T07:13:54.214 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:54.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:54.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:54.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:54.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:54.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:55.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:55.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:55.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:55.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:56.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:56.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:56.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:56.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:57.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:57.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:57.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:57.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:58.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:13:58.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:58.102730+0000) 2019-10-18T07:13:58.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:58.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:58.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:58.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:13:58.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:13:58.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:13:58.852013+0000) 2019-10-18T07:13:59.215 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:13:59.206+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:13:59.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:59.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:13:59.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:13:59.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:00.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:00.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:00.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:00.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:00.891 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:14:00.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:00.889326+0000) 2019-10-18T07:14:01.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:01.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:01.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:01.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:02.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:02.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:02.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:02.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:03.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:03.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:03.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:03.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:04.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:14:04.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:04.110535+0000) 2019-10-18T07:14:04.216 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:04.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:04.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:04.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:04.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:04.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:05.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:05.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:05.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:05.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:06.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:06.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:06.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:06.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:07.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:07.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:07.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:07.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:08.104 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:14:08.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:08.102920+0000) 2019-10-18T07:14:08.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:08.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:08.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:08.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:08.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:14:08.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:08.852119+0000) 2019-10-18T07:14:09.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:09.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:09.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:09.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:09.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:09.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:10.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:10.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:10.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:10.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:10.897 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:14:10.882+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:10.889429+0000) 2019-10-18T07:14:11.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:11.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:11.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:11.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:12.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:12.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:12.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:12.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:13.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:13.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:13.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:13.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:14.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:14:14.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:14.110701+0000) 2019-10-18T07:14:14.217 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:14.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:14.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:14.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:15.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:15.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:15.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:15.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:16.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:16.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:16.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:16.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:17.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:17.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:17.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:17.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:18.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:14:18.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:18.103108+0000) 2019-10-18T07:14:18.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:18.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:18.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:18.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:18.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:14:18.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:18.852226+0000) 2019-10-18T07:14:19.218 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:19.210+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:19.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:19.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:19.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:19.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:20.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:20.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:20.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:20.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:20.891 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:14:20.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:20.889539+0000) 2019-10-18T07:14:21.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:21.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:21.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:21.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:22.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:22.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:22.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:22.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:23.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:23.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:23.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:23.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:24.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:14:24.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:24.110890+0000) 2019-10-18T07:14:24.219 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:24.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:24.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:24.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:24.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:24.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:25.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:25.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:25.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:25.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:26.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:26.489+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:26.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:26.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:27.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:27.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:27.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:27.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:28.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:14:28.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:28.103319+0000) 2019-10-18T07:14:28.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:28.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:28.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:28.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:28.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:14:28.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:28.852333+0000) 2019-10-18T07:14:29.220 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:29.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:29.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:29.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:29.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:29.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:30.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:30.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:30.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:30.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:30.891 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:14:30.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:30.889641+0000) 2019-10-18T07:14:31.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:31.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:31.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:31.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:32.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:32.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:32.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:32.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:33.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:33.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:33.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:33.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:34.112 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:14:34.106+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:34.111065+0000) 2019-10-18T07:14:34.221 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:34.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:34.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:34.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:34.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:34.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:35.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:35.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:35.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:35.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:36.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:36.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:36.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:36.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:37.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:37.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:37.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:37.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:38.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:14:38.098+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:38.103499+0000) 2019-10-18T07:14:38.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:38.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:38.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:38.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:38.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:14:38.845+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:38.852439+0000) 2019-10-18T07:14:39.222 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:39.213+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:39.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:39.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:39.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:39.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:40.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:40.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:40.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:40.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:40.891 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:14:40.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:40.889744+0000) 2019-10-18T07:14:41.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:41.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:41.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:41.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:42.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:42.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:42.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:42.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:43.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:43.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:43.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:43.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:44.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:14:44.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:44.111225+0000) 2019-10-18T07:14:44.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:44.217+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:44.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:44.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:44.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:44.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:45.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:45.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:45.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:45.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:46.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:46.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:46.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:46.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:47.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:47.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:47.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:47.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:48.107 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:14:48.102+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:48.103680+0000) 2019-10-18T07:14:48.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:48.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:48.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:48.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:48.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:14:48.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:48.852545+0000) 2019-10-18T07:14:49.223 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:49.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:49.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:49.521+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:49.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:49.521+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:50.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:50.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:50.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:50.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:50.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:14:50.889+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:50.889844+0000) 2019-10-18T07:14:51.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:51.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:51.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:51.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:52.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:52.569+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:52.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:52.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:53.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:53.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:53.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:53.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:54.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:14:54.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:54.111363+0000) 2019-10-18T07:14:54.224 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:54.221+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:54.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:54.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:04:25.355581+0000 (oldest deadline 2019-10-18T07:04:45.355581+0000) 2019-10-18T07:14:54.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:54.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:55.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:55.653+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:56.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:56.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:57.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:57.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:58.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:14:58.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:58.103865+0000) 2019-10-18T07:14:58.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:58.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:14:58.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:14:58.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:14:58.852653+0000) 2019-10-18T07:14:59.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:14:59.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:14:59.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:14:59.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:00.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:00.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:00.891 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:15:00.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:00.889943+0000) 2019-10-18T07:15:01.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:01.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:02.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:02.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:03.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:03.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:04.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:15:04.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:04.111522+0000) 2019-10-18T07:15:04.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:04.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:04.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:04.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:05.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:05.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:06.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:06.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:07.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:07.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:08.105 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:15:08.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:08.104056+0000) 2019-10-18T07:15:08.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:08.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:08.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:15:08.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:08.852762+0000) 2019-10-18T07:15:09.226 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:09.220+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:09.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:09.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:10.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:10.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:10.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:15:10.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:10.890046+0000) 2019-10-18T07:15:11.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:11.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:12.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:12.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:13.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:13.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:14.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:15:14.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:14.111729+0000) 2019-10-18T07:15:14.228 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:14.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:14.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:14.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:15.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:15.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:16.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:16.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:17.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:17.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:18.106 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:15:18.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:18.104243+0000) 2019-10-18T07:15:18.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:18.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:18.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:18.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:18.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:15:18.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:18.852859+0000) 2019-10-18T07:15:19.229 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:19.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:19.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:19.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:19.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:19.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:20.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:20.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:20.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:20.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:20.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:15:20.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:20.890150+0000) 2019-10-18T07:15:21.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:21.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:21.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:21.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:22.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:22.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:22.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:22.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:23.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:23.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:23.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:23.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:24.113 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:15:24.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:24.111922+0000) 2019-10-18T07:15:24.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:24.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:24.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:24.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:24.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:24.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:25.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:25.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:25.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:25.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:26.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:26.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:26.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:26.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:27.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:27.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:27.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:27.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:28.106 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:15:28.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:28.104374+0000) 2019-10-18T07:15:28.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:28.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:28.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:28.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:28.854 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:15:28.848+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:28.852970+0000) 2019-10-18T07:15:29.230 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:29.224+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:29.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:29.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:29.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:29.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:30.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:30.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:30.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:30.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:30.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:15:30.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:30.890254+0000) 2019-10-18T07:15:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:31.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:31.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:31.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:32.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:32.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:32.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:32.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:33.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:33.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:33.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:33.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:34.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:15:34.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:34.112122+0000) 2019-10-18T07:15:34.231 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:34.223+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:34.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:34.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:34.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:34.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:35.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:35.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:35.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:35.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:36.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:36.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:36.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:36.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:37.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:37.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:37.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:37.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:38.107 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:15:38.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:38.104484+0000) 2019-10-18T07:15:38.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:38.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:38.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:38.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:38.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:15:38.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:38.853076+0000) 2019-10-18T07:15:39.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:39.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:39.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:39.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:39.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:39.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:40.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:40.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:40.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:40.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:40.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:15:40.887+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:40.890359+0000) 2019-10-18T07:15:41.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:41.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:41.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:41.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:42.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:42.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:42.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:42.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:43.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:43.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:43.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:43.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:44.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:15:44.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:44.112321+0000) 2019-10-18T07:15:44.232 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:44.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:44.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:44.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:44.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:44.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:45.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:45.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:45.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:45.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:46.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:46.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:46.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:46.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:47.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:47.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:47.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:47.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:48.106 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:15:48.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:48.104595+0000) 2019-10-18T07:15:48.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:48.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:48.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:48.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:48.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:15:48.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:48.853179+0000) 2019-10-18T07:15:49.234 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:49.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:49.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:49.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:49.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:49.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:50.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:50.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:50.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:50.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:50.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:15:50.887+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:50.890462+0000) 2019-10-18T07:15:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:51.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:51.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:51.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:52.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:52.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:52.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:52.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:53.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:53.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:53.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:53.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:54.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:15:54.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:54.112507+0000) 2019-10-18T07:15:54.234 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:54.227+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:54.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:54.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:54.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:54.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:55.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:55.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:55.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:55.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:56.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:56.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:56.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:56.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:57.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:57.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:57.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:57.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:58.106 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:15:58.101+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:58.104706+0000) 2019-10-18T07:15:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:58.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:58.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:58.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:15:58.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:15:58.847+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:15:58.853284+0000) 2019-10-18T07:15:59.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:15:59.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:15:59.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:59.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:15:59.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:15:59.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:00.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:00.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:00.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:00.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:16:00.887+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:00.890574+0000) 2019-10-18T07:16:01.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:01.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:01.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:01.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:02.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:02.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:02.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:02.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:03.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:03.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:03.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:03.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:04.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:16:04.109+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:04.112718+0000) 2019-10-18T07:16:04.236 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:04.231+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:04.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:04.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:04.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:04.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:05.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:05.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:05.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:05.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:06.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:06.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:06.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:06.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:07.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:07.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:07.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:07.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:08.106 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:08.100+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:08.104816+0000) 2019-10-18T07:16:08.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:08.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:08.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:08.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:08.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:16:08.846+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:08.853390+0000) 2019-10-18T07:16:09.237 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:09.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:09.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:09.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:09.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:09.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:10.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:10.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:10.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:10.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:10.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:16:10.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:10.890681+0000) 2019-10-18T07:16:11.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:11.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:11.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:11.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:12.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:12.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:12.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:12.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:13.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:13.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:13.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:13.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:14.114 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:16:14.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:14.112900+0000) 2019-10-18T07:16:14.238 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:14.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:14.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:14.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:14.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:14.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:15.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:15.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:15.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:15.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:16.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:16.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:16.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:16.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:17.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:17.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:17.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:17.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:18.107 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:18.100+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:18.104931+0000) 2019-10-18T07:16:18.107 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:18.100+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:18.105858+0000) 2019-10-18T07:16:18.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:18.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:18.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:18.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:18.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:16:18.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:18.853491+0000) 2019-10-18T07:16:19.239 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:19.230+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:19.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:19.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:19.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:19.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:20.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:20.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:20.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:20.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:20.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:16:20.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:20.890783+0000) 2019-10-18T07:16:21.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:21.694+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:21.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:21.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:22.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:22.600+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:22.605167+0000) 2019-10-18T07:16:22.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:22.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:22.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:22.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:23.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:23.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:23.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:23.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:24.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:16:24.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:24.113107+0000) 2019-10-18T07:16:24.240 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:24.234+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:24.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:24.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:24.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:24.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:25.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:25.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:25.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:25.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:26.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:26.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:26.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:26.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:27.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:27.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:27.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:27.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:28.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:28.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:28.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:28.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:28.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:16:28.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:28.853596+0000) 2019-10-18T07:16:29.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:29.234+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:29.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:29.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:29.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:29.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:30.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:30.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:30.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:30.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:30.892 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:16:30.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:30.890883+0000) 2019-10-18T07:16:31.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:31.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:31.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:31.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:32.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:32.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:32.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:32.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:32.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:32.600+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:32.605281+0000) 2019-10-18T07:16:33.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:33.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:33.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:33.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:34.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:16:34.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:34.113305+0000) 2019-10-18T07:16:34.241 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:34.234+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:34.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:34.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:34.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:34.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:35.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:35.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:35.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:35.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:36.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:36.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:36.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:36.486+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:37.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:37.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:37.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:37.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:38.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:38.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:38.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:38.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:38.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:16:38.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:38.853701+0000) 2019-10-18T07:16:39.242 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:39.234+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:39.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:39.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:39.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:39.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:40.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:40.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:40.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:40.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:40.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:16:40.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:40.890989+0000) 2019-10-18T07:16:41.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:41.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:41.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:41.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:42.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:42.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:42.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:42.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:42.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:42.600+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:42.605391+0000) 2019-10-18T07:16:43.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:43.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:43.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:43.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:44.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:16:44.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:44.113516+0000) 2019-10-18T07:16:44.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:44.233+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:44.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:44.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:44.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:44.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:45.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:45.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:45.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:45.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:46.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:46.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:46.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:46.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:47.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:47.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:47.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:47.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:48.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:48.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:48.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:16:48.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:48.853801+0000) 2019-10-18T07:16:49.243 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:49.237+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:49.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:49.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:49.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:49.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:50.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:50.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:50.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:50.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:50.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:16:50.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:50.891103+0000) 2019-10-18T07:16:51.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:51.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:51.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:51.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:52.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:52.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:52.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:52.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:52.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:16:52.600+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:52.605501+0000) 2019-10-18T07:16:53.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:53.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:53.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:53.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:54.115 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:16:54.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:54.113705+0000) 2019-10-18T07:16:54.244 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:54.237+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:54.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:54.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:54.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:54.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:55.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:55.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:55.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:55.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:56.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:56.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:56.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:56.589+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:57.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:57.585+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:57.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:57.585+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:58.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:58.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:58.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:58.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:16:58.855 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:16:58.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:16:58.853904+0000) 2019-10-18T07:16:59.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:16:59.237+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:16:59.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:59.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:16:59.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:16:59.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:00.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:00.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:00.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:00.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:17:00.885+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:00.891204+0000) 2019-10-18T07:17:01.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:01.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:01.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:01.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:02.613 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:17:02.600+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:02.605633+0000) 2019-10-18T07:17:02.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:02.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:02.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:02.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:03.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:03.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:03.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:03.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:04.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:17:04.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:04.113882+0000) 2019-10-18T07:17:04.246 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:04.241+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:04.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:04.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:04.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:04.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:05.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:05.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:05.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:05.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:06.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:06.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:06.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:06.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:07.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:07.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:07.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:07.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:08.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:08.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:08.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:08.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:08.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:17:08.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:08.854008+0000) 2019-10-18T07:17:09.247 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:09.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:09.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:09.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:09.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:09.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:10.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:10.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:10.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:10.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:10.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:17:10.889+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:10.891305+0000) 2019-10-18T07:17:11.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:11.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:11.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:11.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:12.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:17:12.600+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:12.605732+0000) 2019-10-18T07:17:12.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:12.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:12.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:12.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:13.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:13.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:13.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:13.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:14.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:17:14.108+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:14.114064+0000) 2019-10-18T07:17:14.248 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:14.245+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:14.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:14.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:14.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:14.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:15.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:15.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:15.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:16.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:16.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:16.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:16.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:17.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:17.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:17.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:17.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:18.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:18.576+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:18.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:18.576+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:18.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:17:18.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:18.854107+0000) 2019-10-18T07:17:19.249 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:19.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:19.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:19.588+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:19.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:19.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:20.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:20.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:20.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:20.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:20.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:17:20.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:20.891404+0000) 2019-10-18T07:17:21.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:21.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:21.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:21.588+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:22.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:22.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:22.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:22.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:17:22.599+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:22.605828+0000) 2019-10-18T07:17:23.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:23.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:23.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:23.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:24.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:17:24.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:24.114246+0000) 2019-10-18T07:17:24.250 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:24.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:24.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:24.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:24.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:24.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:25.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:25.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:25.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:25.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:26.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:26.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:26.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:26.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:27.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:27.548+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:27.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:27.548+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:28.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:28.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:28.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:28.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:28.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:17:28.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:28.854208+0000) 2019-10-18T07:17:29.251 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:29.244+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:29.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:29.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:29.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:29.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:30.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:30.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:30.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:30.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:30.894 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:17:30.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:30.891505+0000) 2019-10-18T07:17:31.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:31.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:31.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:31.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:32.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:32.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:32.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:32.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:32.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:17:32.599+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:32.605930+0000) 2019-10-18T07:17:33.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:33.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:33.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:33.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:34.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:17:34.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:34.114447+0000) 2019-10-18T07:17:34.252 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:34.248+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:34.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:34.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:34.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:34.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:35.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:35.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:35.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:35.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:36.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:36.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:36.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:36.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:37.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:37.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:37.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:37.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:38.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:38.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:38.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:38.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:38.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:17:38.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:38.854308+0000) 2019-10-18T07:17:39.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:39.248+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:39.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:39.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:39.516 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:39.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:40.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:40.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:40.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:40.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:40.896 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:17:40.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:40.891605+0000) 2019-10-18T07:17:41.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:41.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:41.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:41.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:42.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:17:42.599+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:42.606029+0000) 2019-10-18T07:17:42.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:42.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:42.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:42.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:43.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:43.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:43.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:43.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:44.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:17:44.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:44.114659+0000) 2019-10-18T07:17:44.253 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:44.248+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:44.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:44.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:44.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:44.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:45.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:45.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:45.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:45.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:46.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:46.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:46.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:46.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:47.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:47.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:47.630 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:47.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:48.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:48.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:48.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:48.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:48.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:17:48.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:48.854418+0000) 2019-10-18T07:17:49.254 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:49.248+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:49.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:49.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:49.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:49.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:50.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:50.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:50.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:50.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:50.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:17:50.888+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:50.891706+0000) 2019-10-18T07:17:51.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:51.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:51.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:51.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:52.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:52.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:52.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:52.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:52.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:17:52.599+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:52.606125+0000) 2019-10-18T07:17:53.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:53.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:53.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:53.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:54.116 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:17:54.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:54.114868+0000) 2019-10-18T07:17:54.255 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:54.247+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:54.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:54.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:54.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:54.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:55.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:55.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:55.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:55.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:56.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:56.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:56.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:56.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:57.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:57.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:57.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:57.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:58.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:58.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:58.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:58.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:17:58.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:17:58.851+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:17:58.854522+0000) 2019-10-18T07:17:59.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:17:59.251+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:17:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:59.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:17:59.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:17:59.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:00.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:00.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:00.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:00.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:00.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:18:00.887+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:00.891814+0000) 2019-10-18T07:18:01.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:01.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:01.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:01.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:02.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:02.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:02.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:02.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:02.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:18:02.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:02.606227+0000) 2019-10-18T07:18:03.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:03.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:03.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:03.499+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:04.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:18:04.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:04.115056+0000) 2019-10-18T07:18:04.256 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:04.251+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:04.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:04.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:04.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:04.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:05.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:05.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:05.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:05.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:06.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:06.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:06.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:06.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:07.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:07.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:07.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:07.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:08.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:08.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:08.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:08.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:08.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:18:08.851+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:08.854629+0000) 2019-10-18T07:18:09.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:09.251+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:09.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:09.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:09.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:09.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:10.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:10.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:10.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:10.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:10.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:18:10.887+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:10.891922+0000) 2019-10-18T07:18:11.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:11.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:11.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:11.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:12.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:12.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:12.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:12.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:12.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:18:12.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:12.606329+0000) 2019-10-18T07:18:13.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:13.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:13.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:13.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:14.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:18:14.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:14.115242+0000) 2019-10-18T07:18:14.258 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:14.251+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:14.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:14.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:14.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:14.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:15.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:15.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:15.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:16.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:16.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:16.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:16.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:17.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:17.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:17.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:17.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:18.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:18.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:18.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:18.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:18.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:18:18.851+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:18.854730+0000) 2019-10-18T07:18:19.259 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:19.255+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:19.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:19.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:19.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:19.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:20.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:20.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:20.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:20.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:20.893 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:18:20.887+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:20.892024+0000) 2019-10-18T07:18:21.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:21.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:21.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:21.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:22.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:22.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:22.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:22.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:22.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:18:22.603+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:22.606427+0000) 2019-10-18T07:18:23.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:23.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:23.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:23.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:24.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:18:24.111+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:24.115450+0000) 2019-10-18T07:18:24.260 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:24.255+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:24.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:24.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:24.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:24.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:25.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:25.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:25.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:25.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:26.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:26.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:26.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:26.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:27.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:27.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:27.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:27.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:28.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:28.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:28.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:28.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:28.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:18:28.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:28.854831+0000) 2019-10-18T07:18:29.261 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:29.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:29.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:29.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:29.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:29.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:30.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:30.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:30.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:30.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:30.894 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:18:30.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:30.892126+0000) 2019-10-18T07:18:31.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:31.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:31.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:31.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:32.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:32.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:32.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:32.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:32.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:18:32.602+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:32.606526+0000) 2019-10-18T07:18:33.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:33.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:33.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:33.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:34.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:18:34.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:34.115630+0000) 2019-10-18T07:18:34.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:34.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:34.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:34.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:34.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:34.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:35.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:35.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:35.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:35.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:36.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:36.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:36.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:36.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:37.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:37.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:37.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:37.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:38.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:38.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:38.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:38.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:38.856 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:18:38.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:38.854934+0000) 2019-10-18T07:18:39.262 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:39.254+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:39.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:39.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:39.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:39.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:40.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:40.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:40.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:40.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:40.894 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:18:40.886+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:40.892229+0000) 2019-10-18T07:18:41.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:41.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:41.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:41.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:42.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:42.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:42.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:42.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:42.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:18:42.602+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:42.606639+0000) 2019-10-18T07:18:43.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:43.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:43.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:43.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:44.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:18:44.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:44.115822+0000) 2019-10-18T07:18:44.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:44.258+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:44.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:44.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:44.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:44.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:45.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:45.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:45.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:45.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:46.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:46.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:46.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:46.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:47.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:47.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:47.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:47.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:48.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:48.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:48.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:48.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:48.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:18:48.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:48.855038+0000) 2019-10-18T07:18:49.264 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:49.258+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:49.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:49.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:49.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:49.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:50.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:50.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:50.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:50.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:50.895 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:18:50.890+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:50.893647+0000) 2019-10-18T07:18:51.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:51.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:51.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:51.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:52.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:52.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:52.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:52.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:52.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:18:52.602+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:52.606752+0000) 2019-10-18T07:18:53.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:53.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:53.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:54.117 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:18:54.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:54.116034+0000) 2019-10-18T07:18:54.265 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:54.258+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:54.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:54.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:54.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:54.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:55.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:55.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:55.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:55.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:56.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:56.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:56.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:56.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:57.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:57.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:57.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:57.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:58.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:58.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:58.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:58.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:18:58.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:18:58.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:18:58.855143+0000) 2019-10-18T07:18:59.266 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:18:59.258+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:18:59.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:59.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:18:59.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:18:59.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:00.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:00.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:00.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:00.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:00.896 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:19:00.890+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:00.893754+0000) 2019-10-18T07:19:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:01.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:01.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:01.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:02.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:02.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:02.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:02.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:02.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:19:02.602+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:02.606866+0000) 2019-10-18T07:19:03.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:03.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:03.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:03.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:04.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:19:04.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:04.116246+0000) 2019-10-18T07:19:04.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:04.257+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:04.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:04.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:04.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:04.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:05.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:05.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:05.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:05.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:06.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:06.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:06.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:06.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:07.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:07.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:07.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:07.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:08.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:08.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:08.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:08.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:08.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:19:08.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:08.855244+0000) 2019-10-18T07:19:09.267 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:09.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:09.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:09.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:09.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:09.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:10.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:10.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:10.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:10.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:10.895 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:19:10.889+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:10.893856+0000) 2019-10-18T07:19:11.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:11.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:11.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:11.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:12.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:12.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:12.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:12.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:12.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:19:12.602+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:12.606965+0000) 2019-10-18T07:19:13.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:13.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:13.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:13.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:14.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:19:14.110+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:14.116429+0000) 2019-10-18T07:19:14.268 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:14.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:14.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:14.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:14.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:14.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:15.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:15.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:15.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:15.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:16.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:16.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:16.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:16.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:17.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:17.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:17.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:17.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:18.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:18.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:18.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:18.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:18.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:19:18.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:18.855345+0000) 2019-10-18T07:19:19.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:19.261+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:19.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:19.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:19.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:20.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:20.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:20.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:20.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:20.896 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:19:20.889+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:20.893950+0000) 2019-10-18T07:19:21.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:21.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:21.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:21.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:22.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:22.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:22.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:22.321+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:22.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:19:22.602+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:22.607077+0000) 2019-10-18T07:19:23.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:23.293+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:23.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:23.293+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:24.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:19:24.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:24.116614+0000) 2019-10-18T07:19:24.270 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:24.265+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:24.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:24.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:24.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:24.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:25.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:25.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:25.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:25.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:26.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:26.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:26.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:26.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:27.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:27.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:27.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:27.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:28.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:28.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:28.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:28.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:28.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:19:28.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:28.855451+0000) 2019-10-18T07:19:29.272 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:29.265+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:29.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:29.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:29.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:29.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:30.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:30.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:30.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:30.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:30.895 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:19:30.893+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:30.894051+0000) 2019-10-18T07:19:31.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:31.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:31.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:31.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:32.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:32.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:32.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:32.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:32.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:19:32.606+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:32.607194+0000) 2019-10-18T07:19:33.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:33.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:33.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:33.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:34.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:19:34.114+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:34.116796+0000) 2019-10-18T07:19:34.271 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:34.269+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:34.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:34.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:34.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:34.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:35.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:35.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:35.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:35.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:36.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:36.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:36.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:36.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:37.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:37.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:37.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:37.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:38.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:38.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:38.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:38.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:38.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:19:38.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:38.855556+0000) 2019-10-18T07:19:39.273 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:39.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:39.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:39.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:39.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:39.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:40.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:40.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:40.896 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:19:40.892+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:40.894152+0000) 2019-10-18T07:19:41.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:41.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:41.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:41.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:42.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:42.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:42.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:42.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:42.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:19:42.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:42.607308+0000) 2019-10-18T07:19:43.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:43.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:43.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:43.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:44.118 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:19:44.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:44.116976+0000) 2019-10-18T07:19:44.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:44.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:44.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:44.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:44.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:44.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:45.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:45.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:45.493 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:45.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:46.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:46.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:46.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:46.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:47.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:47.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:47.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:47.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:48.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:48.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:48.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:48.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:48.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:19:48.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:48.855659+0000) 2019-10-18T07:19:49.274 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:49.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:49.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:49.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:49.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:50.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:50.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:50.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:50.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:50.899 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:19:50.892+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:50.894251+0000) 2019-10-18T07:19:51.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:51.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:51.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:51.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:52.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:52.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:52.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:52.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:52.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:19:52.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:52.607418+0000) 2019-10-18T07:19:53.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:53.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:53.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:53.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:54.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:19:54.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:54.117156+0000) 2019-10-18T07:19:54.275 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:54.268+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:54.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:54.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:54.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:54.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:55.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:55.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:55.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:55.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:56.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:56.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:56.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:56.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:57.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:57.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:57.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:57.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:58.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:58.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:58.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:58.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:19:58.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:19:58.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:19:58.855764+0000) 2019-10-18T07:19:59.276 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:19:59.272+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:19:59.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:59.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:19:59.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:19:59.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:00.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:00.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:00.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:00.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:00.896 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:00.892+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:00.894354+0000) 2019-10-18T07:20:01.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:01.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:01.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:01.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:02.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:20:02.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:02.607533+0000) 2019-10-18T07:20:02.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:02.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:02.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:02.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:03.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:03.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:03.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:03.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:04.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:20:04.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:04.117337+0000) 2019-10-18T07:20:04.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:04.272+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:04.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:04.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:04.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:04.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:05.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:05.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:05.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:05.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:06.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:06.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:06.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:06.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:07.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:07.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:07.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:07.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:08.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:08.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:08.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:08.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:08.857 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:20:08.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:08.855873+0000) 2019-10-18T07:20:09.277 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:09.272+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:09.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:09.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:09.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:09.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:10.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:10.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:10.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:10.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:10.896 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:10.892+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:10.894451+0000) 2019-10-18T07:20:10.897 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:10.892+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:10.895402+0000) 2019-10-18T07:20:11.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:11.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:11.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:11.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:12.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:20:12.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:12.607648+0000) 2019-10-18T07:20:12.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:12.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:12.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:12.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:13.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:13.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:13.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:13.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:14.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:20:14.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:14.117500+0000) 2019-10-18T07:20:14.278 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:14.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:14.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:14.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:14.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:14.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:15.396 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:15.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:15.394650+0000) 2019-10-18T07:20:15.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:15.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:15.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:15.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:16.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:16.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:16.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:16.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:17.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:17.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:17.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:17.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:18.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:18.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:18.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:18.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:18.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:20:18.851+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:18.855978+0000) 2019-10-18T07:20:19.279 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:19.271+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:19.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:19.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:19.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:19.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:20.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:20.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:20.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:20.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:21.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:21.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:21.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:21.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:22.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:22.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:22.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:22.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:22.609 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:20:22.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:22.607762+0000) 2019-10-18T07:20:23.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:23.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:23.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:23.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:24.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:20:24.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:24.117678+0000) 2019-10-18T07:20:24.280 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:24.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:24.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:24.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:24.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:24.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:25.396 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:25.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:25.394756+0000) 2019-10-18T07:20:25.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:25.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:25.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:25.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:26.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:26.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:26.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:26.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:27.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:27.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:27.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:27.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:28.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:28.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:28.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:28.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:28.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:20:28.851+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:28.856085+0000) 2019-10-18T07:20:29.281 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:29.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:29.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:29.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:29.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:29.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:30.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:30.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:30.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:30.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:31.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:31.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:31.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:31.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:32.611 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:20:32.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:32.607877+0000) 2019-10-18T07:20:32.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:32.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:32.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:32.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:33.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:33.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:33.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:33.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:34.119 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:20:34.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:34.117865+0000) 2019-10-18T07:20:34.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:34.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:34.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:34.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:34.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:34.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:35.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:35.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:35.394853+0000) 2019-10-18T07:20:35.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:35.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:35.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:35.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:36.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:36.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:36.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:36.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:37.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:37.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:37.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:37.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:38.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:38.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:38.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:38.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:38.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:20:38.851+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:38.856184+0000) 2019-10-18T07:20:39.282 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:39.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:39.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:39.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:39.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:39.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:40.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:40.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:40.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:40.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:41.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:41.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:41.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:41.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:42.610 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:20:42.605+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:42.607992+0000) 2019-10-18T07:20:42.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:42.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:42.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:42.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:43.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:43.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:43.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:43.727+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:44.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:20:44.113+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:44.118075+0000) 2019-10-18T07:20:44.283 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:44.275+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:44.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:44.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:44.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:44.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:45.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:45.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:45.394952+0000) 2019-10-18T07:20:45.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:45.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:45.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:45.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:46.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:46.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:46.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:46.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:47.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:47.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:47.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:47.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:48.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:48.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:48.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:48.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:48.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:20:48.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:48.856282+0000) 2019-10-18T07:20:49.284 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:49.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:49.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:49.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:49.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:49.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:50.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:50.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:50.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:50.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:51.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:51.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:51.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:51.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:52.610 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:20:52.604+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:52.608099+0000) 2019-10-18T07:20:52.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:52.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:52.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:52.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:53.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:53.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:53.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:53.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:54.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:20:54.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:54.118253+0000) 2019-10-18T07:20:54.285 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:54.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:54.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:54.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:54.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:54.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:55.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:20:55.390+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:55.395052+0000) 2019-10-18T07:20:55.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:55.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:55.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:55.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:56.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:56.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:56.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:56.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:57.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:57.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:57.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:57.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:58.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:58.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:58.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:58.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:20:58.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:20:58.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:20:58.856390+0000) 2019-10-18T07:20:59.287 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:20:59.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:20:59.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:59.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:20:59.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:20:59.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:00.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:00.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:00.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:00.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:01.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:01.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:01.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:01.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:02.610 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:02.604+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:02.608208+0000) 2019-10-18T07:21:02.611 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:02.604+0000 7f5521f0c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:02.609202+0000) 2019-10-18T07:21:02.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:02.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:02.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:02.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:03.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:03.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:03.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:03.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:04.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:21:04.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:04.118432+0000) 2019-10-18T07:21:04.286 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:04.278+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:04.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:04.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:04.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:04.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:05.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:21:05.390+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:05.395150+0000) 2019-10-18T07:21:05.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:05.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:05.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:05.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:06.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:06.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:06.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:06.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:07.110 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:07.104+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:07.108453+0000) 2019-10-18T07:21:07.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:07.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:07.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:07.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:08.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:08.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:08.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:08.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:08.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:21:08.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:08.856494+0000) 2019-10-18T07:21:09.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:09.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:09.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:09.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:09.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:09.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:10.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:10.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:10.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:10.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 552 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:11.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:11.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:11.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:11.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:12.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:12.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:12.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:12.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:13.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:13.746+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:13.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:13.746+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:14.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:21:14.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:14.118606+0000) 2019-10-18T07:21:14.288 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:14.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:14.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:14.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:14.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:14.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:15.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:21:15.390+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:15.395253+0000) 2019-10-18T07:21:15.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:15.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:15.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:15.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:16.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:16.698+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:16.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:16.698+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:17.110 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:17.104+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:17.108551+0000) 2019-10-18T07:21:17.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:17.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:17.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:17.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:18.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:18.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:18.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:18.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:18.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:21:18.850+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:18.856598+0000) 2019-10-18T07:21:19.290 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:19.282+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:19.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:19.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:19.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:19.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:20.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:20.678+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:20.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:20.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:21.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:21.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:21.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:21.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:22.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:22.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:22.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:22.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:23.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:23.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:23.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:23.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:24.120 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:21:24.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:24.118840+0000) 2019-10-18T07:21:24.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:24.281+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:24.601+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:24.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:24.601+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:25.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:21:25.389+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:25.395358+0000) 2019-10-18T07:21:25.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:25.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:25.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:25.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:26.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:26.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:26.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:26.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:27.110 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:27.104+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:27.108662+0000) 2019-10-18T07:21:27.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:27.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:27.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:27.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:28.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:28.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:28.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:28.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:28.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:21:28.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:28.856706+0000) 2019-10-18T07:21:29.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:29.281+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:29.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:29.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:29.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:29.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:30.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:30.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:30.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:30.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:31.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:31.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:31.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:31.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:32.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:32.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:32.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:32.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:33.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:33.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:33.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:33.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:34.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:21:34.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:34.119051+0000) 2019-10-18T07:21:34.291 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:34.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:34.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:34.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:34.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:34.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:35.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:21:35.389+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:35.395456+0000) 2019-10-18T07:21:35.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:35.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:35.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:35.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:36.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:36.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:36.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:36.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:37.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:37.104+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:37.108773+0000) 2019-10-18T07:21:37.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:37.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:37.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:37.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:38.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:38.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:38.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:38.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:38.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:21:38.849+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:38.856813+0000) 2019-10-18T07:21:39.292 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:39.285+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:39.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:39.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:39.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:39.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:40.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:40.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:40.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:40.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:41.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:41.753+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:41.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:41.753+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:42.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:42.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:42.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:42.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:43.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:43.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:43.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:43.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:44.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:21:44.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:44.119241+0000) 2019-10-18T07:21:44.294 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:44.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:44.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:44.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:44.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:44.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:45.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:21:45.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:45.395569+0000) 2019-10-18T07:21:45.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:45.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:45.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:45.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:46.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:46.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:46.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:46.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:47.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:47.104+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:47.108876+0000) 2019-10-18T07:21:47.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:47.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:47.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:47.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:48.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:48.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:48.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:48.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:48.858 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:21:48.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:48.856918+0000) 2019-10-18T07:21:49.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:49.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:49.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:49.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:49.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:49.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:50.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:50.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:50.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:50.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:51.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:51.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:51.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:51.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:52.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:52.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:52.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:52.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:53.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:53.641+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:53.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:53.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:54.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:21:54.116+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:54.119411+0000) 2019-10-18T07:21:54.295 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:54.289+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:54.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:54.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:54.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:54.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:55.410 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:21:55.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:55.395672+0000) 2019-10-18T07:21:55.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:55.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:55.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:55.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:56.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:56.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:56.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:56.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:57.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:21:57.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:57.108986+0000) 2019-10-18T07:21:57.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:57.620+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:57.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:57.620+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:58.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:58.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:58.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:58.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:21:58.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:21:58.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:21:58.857032+0000) 2019-10-18T07:21:59.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:21:59.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:21:59.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:59.624+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:21:59.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:21:59.624+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:00.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:00.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:00.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:00.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:01.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:01.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:01.603 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:01.596+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:02.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:02.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:02.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:02.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:03.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:03.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:03.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:03.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:04.121 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:04.115+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:04.119581+0000) 2019-10-18T07:22:04.122 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:04.115+0000 7f45249c3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:04.120616+0000) 2019-10-18T07:22:04.296 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:04.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:04.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:04.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:04.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:04.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:05.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:22:05.392+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:05.395770+0000) 2019-10-18T07:22:05.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:05.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:05.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:05.680+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:06.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:06.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:06.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:06.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:07.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:22:07.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:07.109095+0000) 2019-10-18T07:22:07.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:07.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:07.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:07.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:08.622 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:08.615+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:08.620095+0000) 2019-10-18T07:22:08.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:08.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:08.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:08.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:08.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:22:08.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:08.857140+0000) 2019-10-18T07:22:09.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:09.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:09.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:09.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:09.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:09.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:10.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:10.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:10.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:10.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:11.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:11.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:11.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:11.720+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:12.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:12.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:12.721 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:12.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:13.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:13.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:13.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:13.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:14.298 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:14.292+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:14.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:14.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:14.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:14.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:15.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:22:15.392+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:15.395873+0000) 2019-10-18T07:22:15.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:15.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:15.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:15.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:16.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:16.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:16.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:16.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:17.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:22:17.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:17.109208+0000) 2019-10-18T07:22:17.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:17.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:17.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:17.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:18.622 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:18.615+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:18.620274+0000) 2019-10-18T07:22:18.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:22:18.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:18.857242+0000) 2019-10-18T07:22:18.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:18.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:18.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:18.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:19.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:19.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:19.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:19.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:19.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:19.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:20.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:20.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:20.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:20.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:21.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:21.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:21.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:21.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:22.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:22.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:22.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:22.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:23.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:23.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:23.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:23.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:24.300 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:24.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:24.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:24.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:24.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:24.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:25.397 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:22:25.392+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:25.395974+0000) 2019-10-18T07:22:25.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:25.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:25.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:25.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:26.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:26.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:26.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:26.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:27.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:22:27.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:27.109316+0000) 2019-10-18T07:22:27.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:27.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:27.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:27.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:28.622 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:28.615+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:28.620461+0000) 2019-10-18T07:22:28.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:22:28.852+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:28.857349+0000) 2019-10-18T07:22:28.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:28.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:28.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:28.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:29.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:29.296+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:29.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:29.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:29.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:29.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:30.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:30.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:30.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:30.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:32.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:32.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:32.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:32.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:32.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:32.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:32.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:32.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:33.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:33.927+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:33.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:33.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:34.302 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:34.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:34.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:34.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:34.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:34.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:35.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:22:35.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:35.396072+0000) 2019-10-18T07:22:35.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:35.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:35.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:35.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:36.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:36.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:36.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:36.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:37.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:22:37.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:37.109431+0000) 2019-10-18T07:22:37.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:37.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:37.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:37.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:38.622 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:38.615+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:38.620642+0000) 2019-10-18T07:22:38.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:22:38.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:38.857455+0000) 2019-10-18T07:22:38.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:38.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:38.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:38.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:39.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:39.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:39.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:39.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:39.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:39.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:40.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:40.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:40.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:40.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:41.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:41.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:41.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:41.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:42.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:42.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:42.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:42.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:43.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:43.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:43.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:43.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:44.303 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:44.295+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:44.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:44.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:44.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:44.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:45.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:22:45.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:45.396170+0000) 2019-10-18T07:22:45.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:45.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:45.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:45.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:46.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:46.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:46.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:46.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:47.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:22:47.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:47.109551+0000) 2019-10-18T07:22:47.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:47.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:47.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:47.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:48.622 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:48.615+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:48.620825+0000) 2019-10-18T07:22:48.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:48.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:48.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:48.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:48.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:22:48.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:48.857553+0000) 2019-10-18T07:22:49.304 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:49.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:49.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:49.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:49.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:49.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:50.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:50.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:50.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:50.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:51.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:51.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:51.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:51.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:52.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:52.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:52.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:52.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:53.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:53.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:53.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:53.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:54.305 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:54.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:54.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:54.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:54.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:55.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:22:55.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:55.396267+0000) 2019-10-18T07:22:55.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:55.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:55.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:55.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:56.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:56.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:56.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:56.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:57.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:22:57.103+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:57.109667+0000) 2019-10-18T07:22:57.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:57.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:57.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:57.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:58.623 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:22:58.615+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:58.621009+0000) 2019-10-18T07:22:58.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:58.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:58.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:58.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:22:58.859 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:22:58.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:22:58.857657+0000) 2019-10-18T07:22:59.306 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:22:59.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:22:59.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:59.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:22:59.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:22:59.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:00.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:00.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:00.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:00.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:01.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:01.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:01.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:01.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:02.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:02.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:02.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:02.735+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:03.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:03.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:03.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:03.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:04.306 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:04.299+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:04.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:04.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:04.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:04.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:05.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:23:05.391+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:05.396367+0000) 2019-10-18T07:23:05.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:05.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:05.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:05.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:06.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:06.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:06.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:06.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:07.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:23:07.102+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:07.109778+0000) 2019-10-18T07:23:07.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:07.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:07.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:07.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:08.623 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:23:08.614+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:08.621204+0000) 2019-10-18T07:23:08.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:08.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:08.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:08.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:08.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:23:08.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:08.857757+0000) 2019-10-18T07:23:09.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:09.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:09.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:09.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:09.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:09.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:10.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:10.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:10.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:10.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:11.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:11.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:11.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:11.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:12.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:12.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:12.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:12.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:13.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:13.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:13.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:13.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:14.308 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:14.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:14.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:14.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:14.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:14.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:15.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:23:15.390+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:15.396473+0000) 2019-10-18T07:23:15.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:15.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:15.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:15.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:16.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:16.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:16.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:16.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:17.111 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:23:17.102+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:17.109880+0000) 2019-10-18T07:23:17.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:17.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:17.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:17.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:18.623 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:23:18.614+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:18.621408+0000) 2019-10-18T07:23:18.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:18.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:18.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:18.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:18.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:23:18.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:18.857857+0000) 2019-10-18T07:23:19.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:19.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:19.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:19.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:19.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:19.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:20.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:20.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:20.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:20.758+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:21.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:21.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:21.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:21.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:22.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:22.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:22.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:22.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:23.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:23.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:23.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:23.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:24.310 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:24.302+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:24.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:24.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:24.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:24.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:25.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:23:25.390+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:25.396575+0000) 2019-10-18T07:23:25.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:25.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:25.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:25.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:26.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:26.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:26.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:26.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:27.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:23:27.102+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:27.109997+0000) 2019-10-18T07:23:27.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:27.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:27.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:27.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:28.623 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:23:28.614+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:28.621590+0000) 2019-10-18T07:23:28.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:28.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:28.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:28.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:28.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:23:28.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:28.857961+0000) 2019-10-18T07:23:29.311 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:29.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:29.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:29.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:29.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:29.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:30.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:30.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:30.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:30.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:31.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:31.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:31.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:31.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:32.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:32.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:32.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:32.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:33.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:33.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:33.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:33.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:34.312 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:34.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:34.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:34.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:34.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:34.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:35.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:23:35.390+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:35.396679+0000) 2019-10-18T07:23:35.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:35.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:35.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:35.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:36.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:36.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:36.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:36.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:37.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:23:37.102+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:37.110107+0000) 2019-10-18T07:23:37.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:37.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:37.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:37.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:38.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:38.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:38.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:38.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:38.623 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:23:38.614+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:38.621770+0000) 2019-10-18T07:23:38.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:23:38.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:38.858059+0000) 2019-10-18T07:23:39.313 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:39.306+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:39.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:39.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:39.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:39.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:40.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:40.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:40.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:40.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:41.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:41.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:41.604 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:41.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:42.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:42.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:42.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:42.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:43.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:43.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:43.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:43.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:44.314 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:44.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:44.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:44.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:44.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:44.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:45.398 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:23:45.389+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:45.396782+0000) 2019-10-18T07:23:45.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:45.605+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:45.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:45.605+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:46.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:46.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:46.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:46.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:47.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:23:47.106+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:47.110213+0000) 2019-10-18T07:23:47.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:47.589+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:47.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:47.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:48.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:48.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:48.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:48.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:48.624 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:23:48.614+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:48.621951+0000) 2019-10-18T07:23:48.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:23:48.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:48.858158+0000) 2019-10-18T07:23:49.315 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:49.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:49.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:49.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:49.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:49.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:50.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:50.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:50.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:50.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:51.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:51.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:51.645 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:51.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:52.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:52.669+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:52.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:52.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:53.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:53.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:53.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:53.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:54.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:54.305+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:54.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:54.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:54.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:54.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:55.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:23:55.389+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:55.396888+0000) 2019-10-18T07:23:55.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:55.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:55.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:55.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:56.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:56.681+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:56.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:56.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:57.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:23:57.106+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:57.110307+0000) 2019-10-18T07:23:57.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:57.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:57.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:57.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:58.624 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:23:58.614+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:58.622131+0000) 2019-10-18T07:23:58.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:58.709+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:58.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:58.709+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:23:58.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:23:58.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:23:58.858259+0000) 2019-10-18T07:23:59.316 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:23:59.309+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:23:59.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:59.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:23:59.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:23:59.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:00.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:00.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:00.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:00.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:01.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:01.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:01.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:01.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:02.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:02.745+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:02.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:02.745+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:03.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:03.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:03.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:03.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:04.317 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:04.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:04.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:04.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:04.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:04.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:05.399 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:24:05.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:05.396995+0000) 2019-10-18T07:24:05.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:05.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:05.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:05.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:06.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:06.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:06.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:06.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:07.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:24:07.110+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:07.110398+0000) 2019-10-18T07:24:07.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:07.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:07.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:07.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:08.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:24:08.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:08.622349+0000) 2019-10-18T07:24:08.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:08.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:08.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:08.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:08.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:24:08.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:08.858359+0000) 2019-10-18T07:24:09.318 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:09.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:09.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:09.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:09.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:09.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:10.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:10.705+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:10.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:10.705+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:11.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:11.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:11.723 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:11.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:12.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:12.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:12.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:12.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:13.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:13.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:13.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:13.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:14.319 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:14.313+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:14.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:14.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:14.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:14.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:15.399 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:24:15.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:15.397098+0000) 2019-10-18T07:24:15.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:15.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:15.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:15.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:16.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:16.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:16.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:16.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:17.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:24:17.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:17.110492+0000) 2019-10-18T07:24:17.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:17.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:17.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:17.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:18.624 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:24:18.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:18.622530+0000) 2019-10-18T07:24:18.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:18.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:18.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:18.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:18.860 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:24:18.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:18.858461+0000) 2019-10-18T07:24:19.320 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:19.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:19.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:19.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:19.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:19.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:20.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:20.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:20.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:20.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:21.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:21.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:21.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:21.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:22.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:22.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:22.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:22.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:23.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:23.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:23.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:23.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:24.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:24.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:24.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:24.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:24.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:24.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:25.399 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:24:25.392+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:25.397203+0000) 2019-10-18T07:24:25.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:25.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:25.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:25.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:26.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:26.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:26.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:26.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:27.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:24:27.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:27.110601+0000) 2019-10-18T07:24:27.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:27.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:27.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:27.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:28.624 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:24:28.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:28.622716+0000) 2019-10-18T07:24:28.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:24:28.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:28.858565+0000) 2019-10-18T07:24:28.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:28.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:28.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:28.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:29.321 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:29.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:29.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:29.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:29.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:29.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:30.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:30.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:30.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:30.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:31.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:31.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:31.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:31.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:32.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:32.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:32.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:32.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:33.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:33.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:33.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:33.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:34.322 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:34.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:34.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:34.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:34.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:34.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:35.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:24:35.392+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:35.397307+0000) 2019-10-18T07:24:35.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:35.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:35.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:35.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:36.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:36.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:36.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:36.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:37.112 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:24:37.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:37.110693+0000) 2019-10-18T07:24:37.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:37.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:37.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:37.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:38.625 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:24:38.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:38.622921+0000) 2019-10-18T07:24:38.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:24:38.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:38.858666+0000) 2019-10-18T07:24:38.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:38.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:38.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:38.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:39.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:39.316+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:39.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:39.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:39.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:39.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:40.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:40.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:40.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:40.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:42.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:42.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:42.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:42.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:42.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:42.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:42.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:42.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:43.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:43.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:43.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:43.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:44.324 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:44.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:44.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:44.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:44.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:44.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:45.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:24:45.392+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:45.397409+0000) 2019-10-18T07:24:46.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:46.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:46.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:46.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:47.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:46.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:47.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:47.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:47.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:24:47.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:47.110796+0000) 2019-10-18T07:24:47.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:47.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:47.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:47.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:48.625 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:24:48.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:48.623131+0000) 2019-10-18T07:24:48.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:24:48.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:48.858775+0000) 2019-10-18T07:24:48.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:48.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:48.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:48.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:49.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:49.320+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:49.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:49.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:49.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:49.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:50.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:50.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:50.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:50.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:51.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:51.939+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:51.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:51.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:52.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:52.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:52.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:52.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:53.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:53.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:53.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:53.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:54.325 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:54.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:24:54.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:54.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:54.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:54.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:55.399 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:24:55.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:55.397508+0000) 2019-10-18T07:24:56.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:56.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:56.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:56.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:57.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:56.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:57.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:57.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:57.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:24:57.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:57.110905+0000) 2019-10-18T07:24:58.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:58.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:58.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:58.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:58.625 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:24:58.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:58.623312+0000) 2019-10-18T07:24:58.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:24:58.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:24:58.858874+0000) 2019-10-18T07:24:59.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:59.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:24:59.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:24:59.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:24:59.326 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:24:59.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:00.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:00.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:00.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:00.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:01.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:01.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:01.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:01.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:02.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:02.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:02.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:02.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:03.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:03.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:03.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:03.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:04.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:04.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:04.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:04.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:04.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:04.319+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:05.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:05.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:05.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:05.399 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:25:05.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:05.397611+0000) 2019-10-18T07:25:06.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:06.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:06.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:06.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:07.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:07.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:07.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:07.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:07.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:25:07.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:07.111019+0000) 2019-10-18T07:25:08.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:08.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:08.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:08.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:08.625 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:25:08.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:08.623493+0000) 2019-10-18T07:25:08.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:25:08.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:08.858973+0000) 2019-10-18T07:25:09.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:09.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:09.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:09.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:09.328 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:09.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:10.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:10.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:10.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:11.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:11.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:11.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:11.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:12.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:12.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:12.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:12.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:13.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:13.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:13.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:13.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:14.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:14.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:14.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:14.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:14.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:14.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:15.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:15.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:15.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:15.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:15.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:25:15.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:15.397710+0000) 2019-10-18T07:25:16.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:16.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:16.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:17.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:25:17.109+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:17.111134+0000) 2019-10-18T07:25:17.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:17.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:17.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:17.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:18.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:18.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:18.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:18.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:18.626 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:25:18.621+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:18.623671+0000) 2019-10-18T07:25:18.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:25:18.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:18.859069+0000) 2019-10-18T07:25:19.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:19.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:19.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:19.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:19.329 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:19.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:20.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:20.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:20.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:20.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:21.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:21.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:21.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:21.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:22.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:22.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:22.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:22.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:23.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:23.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:23.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:23.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:24.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:24.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:24.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:24.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:24.330 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:24.323+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:25.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:25.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:25.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:25.399 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:25:25.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:25.397802+0000) 2019-10-18T07:25:26.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:26.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:26.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:26.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:27.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:25:27.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:27.111249+0000) 2019-10-18T07:25:27.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:27.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:27.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:27.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:28.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:28.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:28.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:28.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:28.626 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:25:28.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:28.623851+0000) 2019-10-18T07:25:28.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:25:28.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:28.859170+0000) 2019-10-18T07:25:29.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:29.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:29.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:29.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:29.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:29.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:30.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:30.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:30.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:30.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:31.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:31.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:14:58.087983+0000 (oldest deadline 2019-10-18T07:15:18.087983+0000) 2019-10-18T07:25:31.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:31.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:32.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:32.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:33.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:33.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:34.332 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:34.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:34.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:34.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:35.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:35.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:35.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:25:35.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:35.397900+0000) 2019-10-18T07:25:36.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:36.354+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:37.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:25:37.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:37.111363+0000) 2019-10-18T07:25:37.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:37.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:38.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:38.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:38.626 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:25:38.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:38.624042+0000) 2019-10-18T07:25:38.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:25:38.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:38.859270+0000) 2019-10-18T07:25:39.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:39.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:39.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:39.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:40.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:40.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:41.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:41.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:42.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:42.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:43.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:43.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:44.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:44.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:44.333 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:44.326+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:45.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:45.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:45.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:25:45.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:45.398000+0000) 2019-10-18T07:25:46.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:46.226+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:47.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:25:47.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:47.111476+0000) 2019-10-18T07:25:47.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:47.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:48.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:48.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:48.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:25:48.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:48.624232+0000) 2019-10-18T07:25:48.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:25:48.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:48.859381+0000) 2019-10-18T07:25:49.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:49.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:49.335 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:49.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:50.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:50.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:51.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:51.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:52.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:52.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:53.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:53.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:54.336 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:54.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:25:54.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:54.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:55.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:55.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:25:55.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:55.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:55.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:25:55.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:55.398096+0000) 2019-10-18T07:25:56.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:56.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:25:56.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:56.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:57.113 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:25:57.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:57.111592+0000) 2019-10-18T07:25:57.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:57.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:25:57.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:57.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:58.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:58.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:25:58.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:58.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:58.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:25:58.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:58.624418+0000) 2019-10-18T07:25:58.861 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:25:58.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:25:58.859480+0000) 2019-10-18T07:25:59.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:59.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:25:59.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:25:59.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:25:59.337 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:25:59.330+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:00.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:00.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:00.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:00.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:01.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:01.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:01.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:01.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:02.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:02.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:02.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:02.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:03.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:03.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:03.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:03.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:04.339 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:04.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:04.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:04.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:04.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:04.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:05.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:05.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:05.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:05.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:05.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:26:05.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:05.398196+0000) 2019-10-18T07:26:06.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:06.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:06.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:06.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:07.114 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:26:07.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:07.111759+0000) 2019-10-18T07:26:07.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:07.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:07.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:07.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:08.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:08.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:08.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:08.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:08.626 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:26:08.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:08.624597+0000) 2019-10-18T07:26:08.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:26:08.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:08.859575+0000) 2019-10-18T07:26:09.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:09.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:09.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:09.289+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:09.338 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:09.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:10.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:10.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:10.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:10.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:11.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:11.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:11.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:11.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:12.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:12.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:12.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:12.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:13.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:13.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:13.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:13.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:14.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:14.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:14.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:14.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:14.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:14.329+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:15.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:15.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:15.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:15.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:15.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:26:15.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:15.398294+0000) 2019-10-18T07:26:16.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:16.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:16.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:16.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:17.117 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:26:17.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:17.111944+0000) 2019-10-18T07:26:17.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:17.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:17.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:17.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:18.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:18.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:18.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:18.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:18.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:26:18.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:18.624782+0000) 2019-10-18T07:26:18.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:26:18.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:18.859672+0000) 2019-10-18T07:26:19.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:19.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:19.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:19.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:19.340 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:19.333+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:20.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:20.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:20.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:20.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:21.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:21.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:21.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:21.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:22.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:22.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:22.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:22.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:23.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:23.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:23.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:23.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:24.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:24.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:24.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:24.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:24.341 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:24.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:25.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:25.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:25.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:25.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:25.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:26:25.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:25.398412+0000) 2019-10-18T07:26:26.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:26.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:26.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:26.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:27.114 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:26:27.108+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:27.112135+0000) 2019-10-18T07:26:27.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:27.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:27.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:27.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:28.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:28.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:28.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:28.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:28.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:26:28.620+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:28.624988+0000) 2019-10-18T07:26:28.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:26:28.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:28.859776+0000) 2019-10-18T07:26:29.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:29.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:29.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:29.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:29.342 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:29.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:30.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:30.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:30.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:30.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:31.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:31.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:31.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:31.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:32.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:32.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:32.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:32.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:33.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:33.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:33.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:33.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:34.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:34.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:34.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:34.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:34.343 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:34.337+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:35.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:35.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:35.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:35.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:35.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:26:35.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:35.398517+0000) 2019-10-18T07:26:36.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:36.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:36.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:36.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:37.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:37.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:37.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:37.076+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:37.114 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:26:37.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:37.112318+0000) 2019-10-18T07:26:38.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:38.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:38.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:38.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:38.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:26:38.619+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:38.625171+0000) 2019-10-18T07:26:38.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:26:38.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:38.859875+0000) 2019-10-18T07:26:39.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:39.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:39.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:39.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:39.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:39.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:40.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:40.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:40.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:40.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:41.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:41.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:41.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:41.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:42.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:42.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:42.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:42.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:43.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:43.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:43.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:43.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:44.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:44.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:44.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:44.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:44.344 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:44.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:45.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:45.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:45.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:45.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:45.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:26:45.396+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:45.398617+0000) 2019-10-18T07:26:46.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:46.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:46.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:46.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:47.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:47.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:47.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:47.114 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:26:47.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:47.112501+0000) 2019-10-18T07:26:48.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:48.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:48.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:48.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:48.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:26:48.619+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:48.625351+0000) 2019-10-18T07:26:48.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:26:48.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:48.859974+0000) 2019-10-18T07:26:49.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:48.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:49.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:48.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:49.345 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:49.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:50.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:50.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:50.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:50.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:51.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:51.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:51.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:51.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:51.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:51.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:51.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:51.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:52.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:52.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:52.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:53.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:53.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:53.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:53.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:54.346 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:54.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:55.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:55.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:55.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:55.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:55.400 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:26:55.396+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:55.398719+0000) 2019-10-18T07:26:56.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:56.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:56.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:56.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:57.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:57.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:57.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:57.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:57.117 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:26:57.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:57.112715+0000) 2019-10-18T07:26:57.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:57.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:57.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:57.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:58.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:26:58.619+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:58.625535+0000) 2019-10-18T07:26:58.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:26:58.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:26:58.860075+0000) 2019-10-18T07:26:58.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:58.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:58.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:58.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:26:59.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:26:59.340+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:26:59.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:59.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:26:59.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:26:59.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:00.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:00.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:00.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:00.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:01.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:01.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:01.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:01.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:02.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:02.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:02.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:02.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:03.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:03.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:03.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:03.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:04.348 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:04.344+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:04.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:04.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:04.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:04.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:05.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:27:05.396+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:05.398823+0000) 2019-10-18T07:27:05.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:05.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:05.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:05.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:06.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:06.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:06.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:06.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:07.115 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:27:07.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:07.112901+0000) 2019-10-18T07:27:07.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:07.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:07.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:07.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:08.627 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:27:08.619+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:08.625718+0000) 2019-10-18T07:27:08.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:27:08.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:08.860172+0000) 2019-10-18T07:27:08.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:08.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:08.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:08.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:09.349 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:09.344+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:10.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:10.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:10.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:10.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:11.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:11.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:11.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:11.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:12.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:12.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:12.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:12.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:13.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:13.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:13.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:13.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:14.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:14.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:14.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:14.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:14.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:14.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:15.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:15.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:15.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:15.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:15.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:27:15.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:15.398924+0000) 2019-10-18T07:27:16.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:16.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:16.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:16.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:17.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:16.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:17.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:16.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:17.115 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:27:17.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:17.113087+0000) 2019-10-18T07:27:17.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:17.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:17.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:17.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:18.628 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:27:18.619+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:18.625905+0000) 2019-10-18T07:27:18.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:27:18.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:18.860273+0000) 2019-10-18T07:27:19.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:19.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:19.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:19.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:19.350 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:19.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:19.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:19.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:19.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:19.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:20.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:20.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:20.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:20.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:21.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:21.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:21.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:21.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:22.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:22.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:22.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:22.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:23.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:23.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:23.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:23.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:24.351 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:24.343+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:24.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:24.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:24.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:24.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:25.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:27:25.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:25.399036+0000) 2019-10-18T07:27:25.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:25.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:25.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:25.939+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:26.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:26.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:26.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:26.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:27.115 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:27:27.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:27.113272+0000) 2019-10-18T07:27:27.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:27.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:27.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:27.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:28.628 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:27:28.619+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:28.626088+0000) 2019-10-18T07:27:28.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:27:28.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:28.860377+0000) 2019-10-18T07:27:28.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:28.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:28.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:28.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:29.352 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:29.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:29.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:29.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:29.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:29.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:30.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:30.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:30.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:30.871+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:31.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:31.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:31.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:31.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:32.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:32.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:32.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:32.911+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:33.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:33.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:33.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:33.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:34.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:34.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:34.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:34.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:34.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:34.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:35.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:27:35.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:35.399142+0000) 2019-10-18T07:27:35.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:35.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:35.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:35.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:36.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:36.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:36.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:36.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:37.115 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:27:37.107+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:37.113455+0000) 2019-10-18T07:27:37.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:37.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:37.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:37.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:38.628 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:27:38.623+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:38.626270+0000) 2019-10-18T07:27:38.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:38.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:38.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:38.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:38.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:27:38.855+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:38.860477+0000) 2019-10-18T07:27:39.354 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:39.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:39.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:39.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:39.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:39.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:40.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:40.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:40.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:40.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:41.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:41.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:41.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:41.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:42.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:42.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:42.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:42.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:43.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:43.811+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:43.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:43.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:44.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:44.347+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:44.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:44.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:44.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:44.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:45.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:27:45.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:45.399244+0000) 2019-10-18T07:27:45.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:45.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:45.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:45.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:46.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:46.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:46.793 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:46.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:47.115 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:27:47.106+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:47.113641+0000) 2019-10-18T07:27:47.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:47.782+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:47.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:47.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:48.628 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:27:48.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:48.626455+0000) 2019-10-18T07:27:48.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:48.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:48.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:48.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:48.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:27:48.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:48.860580+0000) 2019-10-18T07:27:49.355 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:49.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:49.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:49.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:49.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:49.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:50.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:50.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:50.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:50.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:51.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:51.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:51.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:51.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:52.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:52.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:52.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:52.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:53.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:53.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:53.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:53.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:54.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:54.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:54.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:54.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:54.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:54.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:55.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:27:55.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:55.399350+0000) 2019-10-18T07:27:55.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:55.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:55.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:55.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:56.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:56.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:56.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:56.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:57.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:27:57.106+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:57.113822+0000) 2019-10-18T07:27:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:57.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:57.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:57.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:58.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:27:58.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:58.626637+0000) 2019-10-18T07:27:58.862 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:27:58.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:27:58.860680+0000) 2019-10-18T07:27:58.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:58.906+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:58.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:58.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:27:59.357 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:27:59.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:27:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:59.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:27:59.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:27:59.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:00.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:00.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:00.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:00.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:01.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:01.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:01.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:01.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:02.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:02.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:02.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:02.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:03.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:03.882+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:03.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:03.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:04.358 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:04.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:04.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:04.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:04.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:04.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:05.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:28:05.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:05.399456+0000) 2019-10-18T07:28:05.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:05.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:05.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:05.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:06.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:06.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:06.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:06.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:07.117 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:28:07.106+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:07.114034+0000) 2019-10-18T07:28:07.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:07.886+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:07.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:07.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:08.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:28:08.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:08.626825+0000) 2019-10-18T07:28:08.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:28:08.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:08.860786+0000) 2019-10-18T07:28:08.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:08.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:08.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:08.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:09.359 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:09.350+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:09.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:09.926+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:09.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:09.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:10.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:10.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:10.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:10.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:11.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:11.942+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:11.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:11.942+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:12.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:12.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:12.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:12.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:13.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:13.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:13.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:13.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:14.360 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:14.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:14.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:14.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:14.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:15.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:28:15.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:15.399556+0000) 2019-10-18T07:28:15.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:15.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:15.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:15.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:16.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:16.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:16.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:16.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:17.116 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:28:17.110+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:17.114216+0000) 2019-10-18T07:28:18.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:18.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:18.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:18.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:18.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:28:18.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:18.626970+0000) 2019-10-18T07:28:18.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:28:18.854+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:18.860886+0000) 2019-10-18T07:28:19.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:19.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:19.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:19.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:19.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:19.354+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:20.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:20.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:20.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:20.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:20.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:20.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:21.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:20.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:22.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:21.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:22.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:22.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:22.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:22.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:22.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:22.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:24.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:24.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:24.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:24.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:24.361 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:24.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:25.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:25.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:25.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:25.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:25.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:28:25.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:25.399661+0000) 2019-10-18T07:28:26.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:26.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:26.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:26.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:27.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:27.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:27.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:27.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:27.116 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:28:27.110+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:27.114397+0000) 2019-10-18T07:28:28.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:28.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:28.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:28.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:28.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:28:28.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:28.627152+0000) 2019-10-18T07:28:28.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:28:28.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:28.860987+0000) 2019-10-18T07:28:29.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:29.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:29.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:29.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:29.362 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:29.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:30.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:30.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:30.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:30.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:31.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:31.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:31.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:31.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:32.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:32.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:32.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:32.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:33.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:33.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:33.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:33.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:34.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:34.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:34.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:34.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:34.363 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:34.353+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:35.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:35.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:35.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:35.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:35.401 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:28:35.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:35.399766+0000) 2019-10-18T07:28:36.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:36.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:36.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:36.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:37.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:37.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:37.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:37.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:37.116 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:28:37.110+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:37.114589+0000) 2019-10-18T07:28:38.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:38.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:38.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:38.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:38.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:28:38.622+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:38.627344+0000) 2019-10-18T07:28:38.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:28:38.853+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:38.861090+0000) 2019-10-18T07:28:39.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:39.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:39.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:39.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:39.364 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:39.357+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:40.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:40.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:40.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:40.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:41.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:41.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:41.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:41.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:42.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:42.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:42.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:42.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:43.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:43.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:43.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:43.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:44.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:44.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:44.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:44.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:44.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:44.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:45.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:45.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:45.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:45.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:45.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:28:45.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:45.399866+0000) 2019-10-18T07:28:46.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:46.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:46.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:46.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:47.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:47.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:47.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:47.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:47.117 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:28:47.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:47.114788+0000) 2019-10-18T07:28:48.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:48.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:48.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:48.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:48.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:28:48.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:48.627528+0000) 2019-10-18T07:28:48.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:28:48.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:48.861192+0000) 2019-10-18T07:28:49.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:49.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:49.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:49.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:49.365 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:49.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:50.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:50.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:50.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:50.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:51.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:51.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:51.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:51.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:52.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:52.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:52.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:52.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:53.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:53.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:53.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:53.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:54.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:54.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:54.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:54.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:54.366 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:54.361+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:28:55.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:55.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:55.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:55.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:55.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:28:55.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:55.399974+0000) 2019-10-18T07:28:56.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:56.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:56.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:56.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:57.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:57.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:57.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:57.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:57.117 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:28:57.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:57.115000+0000) 2019-10-18T07:28:58.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:58.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:58.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:58.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:58.629 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:28:58.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:58.627709+0000) 2019-10-18T07:28:58.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:28:58.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:28:58.861292+0000) 2019-10-18T07:28:59.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:59.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:28:59.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:28:59.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:28:59.367 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:28:59.360+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:00.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:00.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:00.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:00.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:01.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:01.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:01.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:01.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:02.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:02.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:02.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:02.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:02.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:02.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:02.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:02.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:03.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:03.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:03.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:03.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:04.368 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:04.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:04.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:04.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:04.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:04.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:05.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:29:05.396+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:05.400080+0000) 2019-10-18T07:29:05.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:05.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:05.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:05.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:06.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:06.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:06.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:06.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:07.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:29:07.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:07.115192+0000) 2019-10-18T07:29:07.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:07.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:07.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:07.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:08.630 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:29:08.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:08.627895+0000) 2019-10-18T07:29:08.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:29:08.856+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:08.861394+0000) 2019-10-18T07:29:08.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:08.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:08.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:08.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:09.369 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:09.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:09.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:09.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:09.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:09.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:10.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:10.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:10.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:10.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:11.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:11.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:11.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:11.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:12.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:12.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:12.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:12.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:13.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:13.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:13.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:13.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:14.370 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:14.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:14.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:14.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:14.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:14.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:15.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:29:15.396+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:15.400187+0000) 2019-10-18T07:29:15.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:15.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:15.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:15.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:16.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:16.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:16.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:16.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:17.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:29:17.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:17.115371+0000) 2019-10-18T07:29:17.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:17.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:17.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:17.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:18.630 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:29:18.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:18.628080+0000) 2019-10-18T07:29:18.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:29:18.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:18.861462+0000) 2019-10-18T07:29:18.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:18.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:18.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:18.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:19.371 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:19.364+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:19.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:19.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:19.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:19.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:21.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:20.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:21.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:20.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:21.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:21.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:21.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:21.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:22.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:22.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:22.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:22.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:23.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:23.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:23.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:23.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:24.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:24.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:24.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:24.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:24.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:24.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:25.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:29:25.396+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:25.400299+0000) 2019-10-18T07:29:25.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:25.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:25.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:25.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:26.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:26.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:26.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:26.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:27.117 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:29:27.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:27.115584+0000) 2019-10-18T07:29:27.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:27.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:27.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:27.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:28.631 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:29:28.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:28.628260+0000) 2019-10-18T07:29:28.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:29:28.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:28.861563+0000) 2019-10-18T07:29:29.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:28.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:29.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:28.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:29.372 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:29.368+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:30.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:29.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:30.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:30.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:31.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:31.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:31.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:31.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:32.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:32.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:32.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:32.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:33.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:33.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:33.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:33.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:34.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:34.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:34.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:34.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:34.373 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:34.367+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:35.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:35.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:35.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:35.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:35.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:29:35.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:35.400405+0000) 2019-10-18T07:29:36.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:36.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:36.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:36.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:36.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:36.991+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:36.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:36.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:37.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:29:37.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:37.115771+0000) 2019-10-18T07:29:38.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:38.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:38.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:38.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:38.630 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:29:38.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:38.628468+0000) 2019-10-18T07:29:38.863 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:29:38.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:38.861663+0000) 2019-10-18T07:29:39.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:39.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:39.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:39.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:39.374 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:39.367+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:40.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:40.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:40.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:40.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:41.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:41.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:41.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:41.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:42.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:42.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:42.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:42.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:43.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:43.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:43.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:43.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:44.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:44.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:44.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:44.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:44.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:44.367+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:45.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:45.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:45.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:45.035+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:45.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:29:45.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:45.400504+0000) 2019-10-18T07:29:46.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:46.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:46.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:46.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:47.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:47.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:47.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:47.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:47.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:29:47.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:47.115988+0000) 2019-10-18T07:29:47.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:47.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:47.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:47.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:48.630 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:29:48.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:48.628646+0000) 2019-10-18T07:29:48.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:29:48.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:48.861760+0000) 2019-10-18T07:29:48.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:48.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:48.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:48.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:49.375 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:49.367+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:50.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:50.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:50.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:50.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:51.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:51.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:51.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:51.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:52.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:52.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:52.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:52.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:53.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:53.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:53.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:53.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:54.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:54.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:54.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:54.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:54.377 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:54.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:29:55.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:55.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:55.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:55.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:55.402 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:29:55.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:55.400613+0000) 2019-10-18T07:29:56.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:56.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:56.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:56.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:57.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:57.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:57.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:57.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:57.119 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:29:57.113+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:57.116171+0000) 2019-10-18T07:29:58.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:58.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:58.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:58.047+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:58.631 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:29:58.625+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:58.628825+0000) 2019-10-18T07:29:58.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:29:58.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:29:58.861860+0000) 2019-10-18T07:29:59.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:59.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:29:59.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:59.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:29:59.378 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:29:59.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:00.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:59.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:00.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:29:59.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:01.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:01.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:01.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:01.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:02.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:02.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:02.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:02.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:03.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:02.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:03.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:02.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:04.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:04.035+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:04.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:04.039+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:04.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:04.371+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:05.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:04.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:05.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:04.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:05.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:30:05.395+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:05.400719+0000) 2019-10-18T07:30:05.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:05.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:05.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:05.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:06.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:06.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:06.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:06.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:07.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:30:07.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:07.116364+0000) 2019-10-18T07:30:08.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:08.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:08.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:08.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:08.631 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:30:08.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:08.628963+0000) 2019-10-18T07:30:08.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:30:08.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:08.861962+0000) 2019-10-18T07:30:08.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:08.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:08.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:08.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:09.379 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:09.370+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:09.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:09.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:09.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:09.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:10.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:10.970+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:10.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:10.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:11.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:11.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:11.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:11.970+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:12.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:12.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:12.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:12.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:14.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:13.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:14.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:13.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:14.380 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:14.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:14.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:14.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:14.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:14.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:15.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:30:15.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:15.400823+0000) 2019-10-18T07:30:15.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:15.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:15.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:15.914+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:16.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:16.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:16.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:16.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:17.118 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:30:17.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:17.116554+0000) 2019-10-18T07:30:17.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:17.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:17.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:17.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:18.633 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:30:18.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:18.629145+0000) 2019-10-18T07:30:18.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:30:18.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:18.862065+0000) 2019-10-18T07:30:18.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:18.866+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:18.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:18.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:19.381 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:19.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:19.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:19.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:19.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:19.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:20.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:20.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:20.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:20.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:21.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:21.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:21.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:21.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:22.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:22.954+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:22.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:22.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:23.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:23.978+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:23.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:23.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:24.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:24.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:24.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:24.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:24.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:24.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:25.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:30:25.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:25.400935+0000) 2019-10-18T07:30:25.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:25.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:25.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:25.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:26.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:26.950+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:26.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:26.954+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:27.119 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:30:27.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:27.116745+0000) 2019-10-18T07:30:28.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:27.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:28.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:27.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:28.632 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:30:28.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:28.629354+0000) 2019-10-18T07:30:28.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:30:28.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:28.862163+0000) 2019-10-18T07:30:28.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:28.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:28.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:28.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:29.382 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:29.374+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:29.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:29.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:29.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:29.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:30.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:30.914+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:30.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:30.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:31.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:31.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:31.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:31.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:32.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:32.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:32.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:32.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:33.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:33.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:33.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:33.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:34.383 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:34.378+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:34.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:34.962+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:34.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:34.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:35.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:30:35.394+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:35.401038+0000) 2019-10-18T07:30:35.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:35.934+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:35.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:35.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:36.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:36.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:36.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:36.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:37.119 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:30:37.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:37.116939+0000) 2019-10-18T07:30:37.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:37.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:37.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:37.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:38.631 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:30:38.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:38.629560+0000) 2019-10-18T07:30:38.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:30:38.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:38.862264+0000) 2019-10-18T07:30:38.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:38.918+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:38.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:38.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:39.384 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:39.378+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:39.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:39.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:39.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:39.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:40.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:40.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:40.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:40.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:41.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:41.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:41.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:41.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:42.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:42.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:42.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:42.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:44.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:44.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:44.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:44.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:44.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:44.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:44.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:44.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:44.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:44.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:45.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:30:45.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:45.401131+0000) 2019-10-18T07:30:45.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:45.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:45.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:45.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:46.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:46.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:46.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:46.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:47.119 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:30:47.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:47.117119+0000) 2019-10-18T07:30:47.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:47.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:47.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:47.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:48.637 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:30:48.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:48.629735+0000) 2019-10-18T07:30:48.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:30:48.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:48.862360+0000) 2019-10-18T07:30:48.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:48.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:48.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:48.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:49.385 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:49.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:49.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:49.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:49.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:49.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:50.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:50.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:50.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:50.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:51.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:51.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:51.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:51.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:53.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:53.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:53.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:53.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:54.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:54.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:54.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:54.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:54.387 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:54.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:30:55.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:54.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:55.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:54.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:55.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:30:55.393+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:55.401233+0000) 2019-10-18T07:30:56.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:56.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:56.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:56.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:57.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:57.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:57.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:57.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:57.119 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:30:57.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:57.117301+0000) 2019-10-18T07:30:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:58.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:58.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:58.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:58.632 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:30:58.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:58.629915+0000) 2019-10-18T07:30:58.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:30:58.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:30:58.862461+0000) 2019-10-18T07:30:59.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:59.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:30:59.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:30:59.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:30:59.388 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:30:59.377+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:00.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:00.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:00.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:00.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:01.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:01.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:01.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:01.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:02.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:02.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:02.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:02.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:03.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:03.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:03.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:03.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:04.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:04.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:04.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:04.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:04.389 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:04.385+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:05.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:05.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:05.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:05.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:05.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:31:05.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:05.401352+0000) 2019-10-18T07:31:06.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:06.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:06.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:06.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:07.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:07.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:07.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:07.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:07.120 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:31:07.112+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:07.117487+0000) 2019-10-18T07:31:08.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:08.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:08.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:08.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:08.632 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:31:08.624+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:08.630152+0000) 2019-10-18T07:31:08.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:31:08.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:08.862562+0000) 2019-10-18T07:31:09.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:09.173+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:09.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:09.173+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:09.390 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:09.385+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:10.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:10.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:10.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:10.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:11.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:11.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:11.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:11.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:12.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:12.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:12.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:12.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:13.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:13.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:13.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:13.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:14.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:14.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:14.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:14.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:14.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:14.385+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:15.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:15.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:15.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:15.221+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:15.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:31:15.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:15.401458+0000) 2019-10-18T07:31:16.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:16.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:16.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:16.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:17.120 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:31:17.111+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:17.117705+0000) 2019-10-18T07:31:17.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:17.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:17.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:17.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:18.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:18.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:18.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:18.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:18.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:31:18.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:18.630356+0000) 2019-10-18T07:31:18.864 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:31:18.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:18.862665+0000) 2019-10-18T07:31:19.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:19.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:19.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:19.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:19.391 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:19.384+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:20.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:20.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:20.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:20.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:21.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:21.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:21.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:21.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:22.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:22.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:22.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:22.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:23.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:23.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:23.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:23.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:24.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:24.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:24.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:24.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:24.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:24.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:25.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:25.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:25.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:25.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:25.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:31:25.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:25.401562+0000) 2019-10-18T07:31:26.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:26.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:26.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:26.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:27.120 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:31:27.111+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:27.117917+0000) 2019-10-18T07:31:27.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:27.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:27.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:27.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:28.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:28.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:28.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:28.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:28.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:31:28.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:28.630566+0000) 2019-10-18T07:31:28.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:31:28.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:28.862763+0000) 2019-10-18T07:31:29.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:29.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:29.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:29.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:29.392 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:29.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:30.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:30.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:30.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:30.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:31.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:31.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:31.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:31.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:32.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:32.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:32.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:32.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:33.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:33.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:33.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:33.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:34.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:34.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:34.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:34.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:34.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:34.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:35.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:31:35.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:35.401657+0000) 2019-10-18T07:31:35.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:35.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:35.457 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:35.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:36.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:36.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:36.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:36.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:37.122 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:31:37.111+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:37.118131+0000) 2019-10-18T07:31:37.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:37.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:37.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:37.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:38.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:38.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:38.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:38.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:38.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:31:38.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:38.630774+0000) 2019-10-18T07:31:38.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:31:38.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:38.862865+0000) 2019-10-18T07:31:39.395 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:39.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:39.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:39.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:39.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:39.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:40.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:40.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:40.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:40.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:41.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:41.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:41.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:41.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:42.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:42.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:42.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:42.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:43.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:43.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:43.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:43.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:44.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:44.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:44.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:44.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:44.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:44.388+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:45.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:45.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:45.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:45.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:45.403 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:31:45.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:45.401752+0000) 2019-10-18T07:31:46.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:46.400+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:46.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:46.404+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:47.120 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:31:47.115+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:47.118371+0000) 2019-10-18T07:31:47.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:47.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:47.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:47.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:48.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:48.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:48.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:48.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:48.633 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:31:48.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:48.630961+0000) 2019-10-18T07:31:48.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:31:48.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:48.862967+0000) 2019-10-18T07:31:49.396 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:49.392+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:49.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:49.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:49.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:49.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:50.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:50.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:50.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:50.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:51.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:51.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:51.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:51.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:52.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:52.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:52.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:52.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:53.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:53.475+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:53.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:53.475+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:54.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:54.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:54.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:54.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:54.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:54.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:55.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:31:55.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:55.401858+0000) 2019-10-18T07:31:55.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:55.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:55.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:55.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:56.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:56.479+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:56.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:56.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:57.120 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:31:57.115+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:57.118553+0000) 2019-10-18T07:31:57.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:57.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:57.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:57.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:58.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:58.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:58.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:58.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:31:58.633 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:31:58.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:58.631168+0000) 2019-10-18T07:31:58.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:31:58.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:31:58.863070+0000) 2019-10-18T07:31:59.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:31:59.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:31:59.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:59.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:31:59.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:31:59.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:00.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:00.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:00.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:00.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:01.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:01.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:01.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:01.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:02.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:02.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:02.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:02.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:03.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:03.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:03.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:03.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:04.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:04.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:04.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:04.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:04.398 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:04.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:05.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:32:05.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:05.401961+0000) 2019-10-18T07:32:05.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:05.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:05.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:05.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:06.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:06.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:06.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:06.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:07.121 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:32:07.115+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:07.118763+0000) 2019-10-18T07:32:07.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:07.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:07.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:07.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:08.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:08.371+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:08.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:08.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:08.633 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:32:08.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:08.631380+0000) 2019-10-18T07:32:08.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:32:08.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:08.863172+0000) 2019-10-18T07:32:09.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:09.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:09.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:09.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:09.399 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:09.391+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:10.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:10.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:10.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:10.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:11.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:11.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:11.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:11.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:12.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:12.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:12.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:12.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:13.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:13.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:13.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:13.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:14.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:14.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:14.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:14.400 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:14.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:15.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:15.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:15.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:15.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:15.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:32:15.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:15.402055+0000) 2019-10-18T07:32:16.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:16.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:16.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:16.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:17.121 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:32:17.115+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:17.118940+0000) 2019-10-18T07:32:17.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:17.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:17.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:17.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:18.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:18.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:18.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:18.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:18.633 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:32:18.627+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:18.631600+0000) 2019-10-18T07:32:18.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:32:18.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:18.863276+0000) 2019-10-18T07:32:19.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:19.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:19.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:19.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:19.401 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:19.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:20.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:20.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:20.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:20.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:21.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:21.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:21.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:21.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:22.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:22.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:22.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:22.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:23.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:23.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:23.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:23.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:24.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:24.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:24.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:24.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:24.402 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:24.395+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:25.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:25.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:25.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:25.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:32:25.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:25.402161+0000) 2019-10-18T07:32:26.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:26.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:26.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:26.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:27.121 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:32:27.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:27.119120+0000) 2019-10-18T07:32:27.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:27.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:27.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:27.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:28.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:28.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:28.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:28.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:28.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:32:28.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:28.631784+0000) 2019-10-18T07:32:28.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:32:28.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:28.863379+0000) 2019-10-18T07:32:29.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:29.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:29.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:29.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:29.403 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:29.394+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:30.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:30.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:30.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:30.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:31.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:31.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:31.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:31.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:32.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:32.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:32.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:32.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:33.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:33.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:33.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:33.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:34.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:34.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:34.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:34.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:34.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:34.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:35.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:35.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:35.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:35.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:35.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:32:35.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:35.402263+0000) 2019-10-18T07:32:36.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:36.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:36.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:36.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:37.121 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:32:37.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:37.119332+0000) 2019-10-18T07:32:37.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:37.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:37.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:37.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:38.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:38.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:38.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:38.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:38.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:32:38.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:38.631987+0000) 2019-10-18T07:32:38.865 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:32:38.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:38.863479+0000) 2019-10-18T07:32:39.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:39.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:39.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:39.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:39.404 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:39.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:40.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:40.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:40.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:40.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:41.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:41.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:41.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:41.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:42.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:42.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:42.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:42.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:43.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:43.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:43.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:43.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:44.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:44.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:44.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:44.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:44.406 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:44.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:45.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:45.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:45.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:45.258+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:45.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:32:45.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:45.402365+0000) 2019-10-18T07:32:46.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:46.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:46.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:46.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:47.121 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:32:47.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:47.119544+0000) 2019-10-18T07:32:47.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:47.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:47.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:47.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:48.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:48.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:48.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:48.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:48.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:32:48.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:48.632188+0000) 2019-10-18T07:32:48.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:32:48.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:48.863582+0000) 2019-10-18T07:32:49.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:49.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:49.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:49.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:49.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:49.398+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:50.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:50.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:50.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:50.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:51.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:51.234+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:51.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:51.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:52.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:52.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:52.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:52.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:53.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:53.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:53.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:53.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:54.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:54.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:54.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:54.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:54.408 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:54.402+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:32:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:55.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:55.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:55.234+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:55.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:32:55.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:55.402474+0000) 2019-10-18T07:32:56.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:56.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:56.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:56.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:57.122 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:32:57.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:57.119726+0000) 2019-10-18T07:32:57.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:57.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:57.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:57.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:58.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:58.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:58.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:58.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:58.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:32:58.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:58.632393+0000) 2019-10-18T07:32:58.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:32:58.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:32:58.863683+0000) 2019-10-18T07:32:59.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:59.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:32:59.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:32:59.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:32:59.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:32:59.402+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:00.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:00.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:00.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:00.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:01.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:01.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:01.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:01.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:02.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:02.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:02.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:02.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:03.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:03.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:03.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:03.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:04.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:04.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:04.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:04.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:04.409 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:04.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:05.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:05.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:05.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:05.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:05.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:33:05.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:05.402583+0000) 2019-10-18T07:33:06.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:06.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:06.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:06.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:07.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:33:07.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:07.119937+0000) 2019-10-18T07:33:07.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:07.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:07.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:07.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:08.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:08.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:08.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:08.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:08.634 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:33:08.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:08.632605+0000) 2019-10-18T07:33:08.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:33:08.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:08.863785+0000) 2019-10-18T07:33:09.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:09.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:09.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:09.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:09.410 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:09.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:10.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:10.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:10.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:10.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:11.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:11.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:11.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:11.273+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:12.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:12.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:12.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:12.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:13.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:13.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:13.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:13.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:14.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:14.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:14.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:14.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:14.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:14.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:15.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:15.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:15.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:15.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:15.404 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:33:15.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:15.402690+0000) 2019-10-18T07:33:16.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:16.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:16.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:16.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:17.122 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:33:17.114+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:17.120119+0000) 2019-10-18T07:33:17.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:17.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:17.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:17.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:18.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:18.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:18.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:18.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:18.635 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:33:18.626+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:18.632789+0000) 2019-10-18T07:33:18.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:33:18.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:18.863887+0000) 2019-10-18T07:33:19.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:19.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:19.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:19.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:19.411 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:19.401+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:20.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:20.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:20.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:20.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:21.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:21.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:21.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:21.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:22.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:22.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:22.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:22.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:23.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:23.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:23.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:23.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:24.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:24.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:24.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:24.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:24.413 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:24.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:25.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:25.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:25.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:25.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:25.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:33:25.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:25.402798+0000) 2019-10-18T07:33:26.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:26.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:26.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:26.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:27.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:33:27.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:27.120311+0000) 2019-10-18T07:33:27.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:27.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:27.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:27.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:28.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:28.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:28.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:28.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:28.635 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:33:28.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:28.632982+0000) 2019-10-18T07:33:28.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:33:28.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:28.863989+0000) 2019-10-18T07:33:29.414 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:29.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:29.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:29.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:29.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:29.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:30.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:30.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:30.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:30.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:31.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:31.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:31.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:31.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:32.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:32.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:32.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:33.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:33.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:33.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:33.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:34.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:34.409+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:34.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:34.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:34.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:34.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:35.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:33:35.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:35.402909+0000) 2019-10-18T07:33:35.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:35.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:35.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:35.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:36.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:36.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:36.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:36.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:37.122 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:33:37.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:37.120519+0000) 2019-10-18T07:33:37.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:37.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:37.403 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:37.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:38.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:38.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:38.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:38.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:38.635 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:33:38.629+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:38.633194+0000) 2019-10-18T07:33:38.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:33:38.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:38.864090+0000) 2019-10-18T07:33:39.415 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:39.408+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:39.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:39.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:39.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:39.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:40.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:40.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:40.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:40.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:41.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:41.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:41.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:42.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:42.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:42.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:42.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:43.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:43.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:43.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:43.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:44.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:44.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:44.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:44.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:44.416 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:44.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:45.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:45.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:45.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:45.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:45.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:33:45.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:45.403013+0000) 2019-10-18T07:33:46.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:46.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:46.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:46.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:47.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:33:47.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:47.120732+0000) 2019-10-18T07:33:47.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:47.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:47.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:47.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:48.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:48.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:48.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:48.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:48.635 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:33:48.629+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:48.633410+0000) 2019-10-18T07:33:48.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:33:48.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:48.864191+0000) 2019-10-18T07:33:49.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:49.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:49.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:49.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:49.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:49.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:50.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:50.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:50.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:50.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:51.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:51.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:51.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:51.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:52.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:52.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:52.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:52.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:53.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:53.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:53.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:53.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:54.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:54.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:54.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:54.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:54.417 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:54.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:33:55.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:55.148+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:55.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:55.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:55.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:33:55.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:55.403115+0000) 2019-10-18T07:33:56.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:56.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:56.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:56.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:57.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:33:57.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:57.120887+0000) 2019-10-18T07:33:57.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:57.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:57.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:57.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:58.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:58.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:58.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:58.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:58.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:33:58.629+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:58.633618+0000) 2019-10-18T07:33:58.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:33:58.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:33:58.864286+0000) 2019-10-18T07:33:59.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:59.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:33:59.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:33:59.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:33:59.419 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:33:59.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:00.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:00.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:00.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:00.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:01.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:01.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:01.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:01.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:02.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:02.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:02.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:02.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:03.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:03.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:03.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:03.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:04.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:04.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:04.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:04.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:04.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:04.412+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:05.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:05.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:05.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:05.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:05.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:34:05.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:05.403223+0000) 2019-10-18T07:34:06.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:06.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:06.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:06.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:07.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:34:07.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:07.121003+0000) 2019-10-18T07:34:07.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:07.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:07.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:07.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:08.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:08.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:08.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:08.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:08.636 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:34:08.629+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:08.633780+0000) 2019-10-18T07:34:08.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:34:08.860+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:08.864381+0000) 2019-10-18T07:34:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:09.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:09.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:09.420 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:09.416+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:10.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:10.184+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:10.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:10.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:11.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:11.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:11.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:11.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:12.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:12.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:12.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:12.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:13.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:13.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:13.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:13.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:14.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:14.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:14.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:14.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:14.421 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:14.415+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:15.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:15.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:15.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:15.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:15.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:34:15.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:15.403329+0000) 2019-10-18T07:34:16.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:16.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:16.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:16.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:17.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:34:17.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:17.121111+0000) 2019-10-18T07:34:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:17.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:17.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:17.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:18.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:18.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:18.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:18.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:18.636 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:34:18.629+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:18.634001+0000) 2019-10-18T07:34:18.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:34:18.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:18.864479+0000) 2019-10-18T07:34:19.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:19.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:19.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:19.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:19.422 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:19.415+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:20.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:20.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:21.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:21.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:21.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:21.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:22.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:22.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:22.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:22.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:23.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:23.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:23.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:23.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:24.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:24.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:24.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:24.415+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:25.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:25.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:25.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:25.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:25.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:34:25.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:25.403433+0000) 2019-10-18T07:34:26.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:26.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:26.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:26.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:27.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:34:27.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:27.121223+0000) 2019-10-18T07:34:27.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:27.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:27.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:27.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:28.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:28.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:28.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:28.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:28.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:34:28.633+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:28.634202+0000) 2019-10-18T07:34:28.866 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:34:28.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:28.864579+0000) 2019-10-18T07:34:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:29.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:29.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:29.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:29.423 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:29.415+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:30.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:30.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:30.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:30.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:31.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:31.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:31.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:31.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:32.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:32.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:32.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:32.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:33.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:33.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:33.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:33.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:34.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:34.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:34.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:34.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:34.425 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:34.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:35.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:35.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:35.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:35.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:35.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:34:35.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:35.403538+0000) 2019-10-18T07:34:36.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:36.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:36.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:36.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:37.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:34:37.117+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:37.121334+0000) 2019-10-18T07:34:37.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:37.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:37.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:37.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:38.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:38.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:38.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:38.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:38.636 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:34:38.633+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:38.634350+0000) 2019-10-18T07:34:38.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:34:38.859+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:38.864681+0000) 2019-10-18T07:34:39.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:39.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:39.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:39.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:39.426 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:39.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:40.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:40.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:40.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:40.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:41.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:41.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:41.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:41.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:42.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:42.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:42.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:42.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:43.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:43.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:43.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:43.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:44.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:44.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:44.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:44.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:44.427 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:44.419+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:45.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:45.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:45.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:45.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:45.405 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:34:45.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:45.403646+0000) 2019-10-18T07:34:46.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:46.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:46.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:46.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:47.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:34:47.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:47.121444+0000) 2019-10-18T07:34:47.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:47.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:47.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:47.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:48.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:48.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:48.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:48.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:48.636 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:34:48.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:48.634501+0000) 2019-10-18T07:34:48.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:34:48.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:48.864778+0000) 2019-10-18T07:34:49.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:49.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:49.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:49.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:49.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:49.418+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:50.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:50.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:50.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:50.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:51.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:51.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:51.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:51.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:52.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:52.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:52.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:52.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:53.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:53.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:53.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:53.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:54.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:54.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:54.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:54.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:54.428 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:54.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:34:55.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:55.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:55.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:55.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:55.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:34:55.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:55.403749+0000) 2019-10-18T07:34:56.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:56.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:56.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:56.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:57.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:34:57.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:57.121551+0000) 2019-10-18T07:34:57.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:57.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:57.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:57.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:58.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:58.246+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:58.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:58.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:58.637 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:34:58.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:58.634665+0000) 2019-10-18T07:34:58.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:34:58.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:34:58.864879+0000) 2019-10-18T07:34:59.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:59.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:34:59.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:34:59.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:34:59.429 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:34:59.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:00.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:00.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:00.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:00.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:01.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:01.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:01.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:01.246+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:02.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:02.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:02.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:02.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:03.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:03.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:03.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:04.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:04.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:04.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:04.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:04.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:04.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:05.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:05.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:05.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:05.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:05.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:35:05.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:05.403852+0000) 2019-10-18T07:35:06.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:06.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:06.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:06.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:07.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:35:07.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:07.121665+0000) 2019-10-18T07:35:07.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:07.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:07.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:07.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:08.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:08.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:08.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:08.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:08.637 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:35:08.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:08.634832+0000) 2019-10-18T07:35:08.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:35:08.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:08.864981+0000) 2019-10-18T07:35:09.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:09.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:09.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:09.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:09.431 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:09.422+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:10.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:10.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:10.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:10.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:11.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:11.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:11.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:11.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:11.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:11.982+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:11.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:11.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:13.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:13.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:13.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:13.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:14.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:13.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:14.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:13.994+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:14.432 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:14.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:15.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:15.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:15.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:15.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:15.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:35:15.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:15.403954+0000) 2019-10-18T07:35:16.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:16.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:16.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:16.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:17.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:17.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:17.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:17.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:17.123 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:35:17.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:17.121778+0000) 2019-10-18T07:35:18.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:18.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:18.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:18.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:18.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:35:18.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:18.635013+0000) 2019-10-18T07:35:18.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:35:18.858+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:18.865076+0000) 2019-10-18T07:35:19.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:19.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:19.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:19.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:19.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:19.426+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:20.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:20.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:20.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:20.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:21.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:21.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:21.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:21.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:22.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:22.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:22.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:22.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:23.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:23.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:23.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:23.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:24.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:24.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:24.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:24.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:24.433 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:24.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:25.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:25.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:25.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:25.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:25.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:35:25.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:25.404061+0000) 2019-10-18T07:35:26.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:26.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:26.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:26.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:27.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:35:27.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:27.121888+0000) 2019-10-18T07:35:27.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:27.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:27.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:27.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:28.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:28.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:28.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:28.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:28.637 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:35:28.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:28.635192+0000) 2019-10-18T07:35:28.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:35:28.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:28.865174+0000) 2019-10-18T07:35:29.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:29.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:29.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:29.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:29.434 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:29.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:30.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:30.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:30.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:30.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:31.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:31.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:31.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:31.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:32.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:32.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:32.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:32.113+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:33.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:33.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:33.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:33.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:34.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:34.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:34.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:34.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:34.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:34.425+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:35.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:35.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:35.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:35.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:35.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:35:35.397+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:35.404168+0000) 2019-10-18T07:35:36.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:36.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:36.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:36.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:37.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:37.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:37.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:37.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:37.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:35:37.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:37.121995+0000) 2019-10-18T07:35:38.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:38.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:38.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:38.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:38.637 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:35:38.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:38.635371+0000) 2019-10-18T07:35:38.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:35:38.857+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:38.865277+0000) 2019-10-18T07:35:39.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:39.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:39.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:39.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:39.436 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:39.429+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:40.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:40.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:40.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:40.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:41.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:41.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:41.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:41.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:42.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:42.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:42.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:42.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:43.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:43.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:43.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:43.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:44.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:44.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:44.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:44.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:44.437 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:44.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:45.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:45.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:45.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:45.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:45.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:35:45.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:45.404288+0000) 2019-10-18T07:35:46.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:46.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:46.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:46.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:47.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:47.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:47.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:47.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:47.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:35:47.116+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:47.122105+0000) 2019-10-18T07:35:48.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:48.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:48.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:48.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:48.637 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:35:48.632+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:48.635585+0000) 2019-10-18T07:35:48.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:35:48.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:48.865379+0000) 2019-10-18T07:35:49.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:49.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:49.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:49.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:49.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:49.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:50.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:50.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:50.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:50.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:51.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:51.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:51.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:51.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:51.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:51.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:51.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:51.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:52.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:52.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:52.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:52.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:53.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:53.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:53.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:53.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:54.438 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:54.433+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:55.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:55.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:55.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:55.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:55.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:35:55.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:55.404397+0000) 2019-10-18T07:35:56.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:56.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:56.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:56.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:57.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:57.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:57.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:57.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:35:57.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:57.122218+0000) 2019-10-18T07:35:58.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:57.996+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:58.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:57.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:58.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:35:58.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:58.635765+0000) 2019-10-18T07:35:58.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:35:58.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:35:58.865476+0000) 2019-10-18T07:35:59.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:59.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:59.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:59.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:35:59.440 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:35:59.432+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:35:59.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:59.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:35:59.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:35:59.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:00.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:00.976+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:36:00.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:00.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:01.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:01.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:36:01.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:01.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:02.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:02.956+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:36:02.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:02.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:03.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:03.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:25:35.023427+0000 (oldest deadline 2019-10-18T07:25:55.023427+0000) 2019-10-18T07:36:04.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:03.992+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:04.444 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:04.436+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:05.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:05.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:05.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:36:05.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:05.404504+0000) 2019-10-18T07:36:05.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:05.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:06.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:06.976+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:07.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:36:07.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:07.122313+0000) 2019-10-18T07:36:07.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:07.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:08.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:36:08.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:08.635948+0000) 2019-10-18T07:36:08.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:36:08.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:08.865581+0000) 2019-10-18T07:36:08.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:08.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:09.441 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:09.436+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:09.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:09.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:10.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:10.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 564 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:11.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:11.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:12.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:12.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:13.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:13.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:14.442 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:14.436+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:14.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:14.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:15.406 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:36:15.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:15.404615+0000) 2019-10-18T07:36:15.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:15.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:16.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:16.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:17.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:36:17.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:17.122408+0000) 2019-10-18T07:36:17.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:17.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:18.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:36:18.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:18.636124+0000) 2019-10-18T07:36:18.867 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:36:18.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:18.865684+0000) 2019-10-18T07:36:18.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:18.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:19.444 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:19.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:19.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:19.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:20.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:20.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:21.934 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:21.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:22.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:22.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:23.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:23.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:24.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:24.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:24.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:24.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:25.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:36:25.400+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:25.404725+0000) 2019-10-18T07:36:25.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:25.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:25.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:25.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:26.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:26.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:26.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:26.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:27.124 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:36:27.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:27.122505+0000) 2019-10-18T07:36:27.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:27.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:27.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:27.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:28.639 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:36:28.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:28.636303+0000) 2019-10-18T07:36:28.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:36:28.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:28.865781+0000) 2019-10-18T07:36:28.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:28.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:28.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:28.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:29.445 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:29.440+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:29.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:29.872+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:29.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:29.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:30.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:30.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:30.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:30.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:31.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:31.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:31.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:31.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:32.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:32.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:32.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:32.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:33.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:33.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:33.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:33.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:34.446 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:34.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:34.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:34.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:34.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:34.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:35.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:36:35.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:35.404833+0000) 2019-10-18T07:36:35.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:35.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:35.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:35.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:36.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:36.979+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:36.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:36.979+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:37.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:36:37.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:37.122605+0000) 2019-10-18T07:36:38.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:38.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:38.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:38.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:38.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:36:38.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:38.636513+0000) 2019-10-18T07:36:38.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:36:38.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:38.865879+0000) 2019-10-18T07:36:39.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:39.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:39.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:39.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:39.447 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:39.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:40.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:40.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:40.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:40.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:41.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:41.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:41.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:41.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:42.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:42.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:42.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:42.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:43.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:43.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:43.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:43.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:44.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:44.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:44.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:44.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:44.448 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:44.439+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:45.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:45.043+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:45.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:45.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:45.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:36:45.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:45.404935+0000) 2019-10-18T07:36:46.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:46.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:46.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:46.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:47.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:47.039+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:47.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:47.043+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:47.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:36:47.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:47.122719+0000) 2019-10-18T07:36:48.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:48.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:48.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:48.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:48.638 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:36:48.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:48.636693+0000) 2019-10-18T07:36:48.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:36:48.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:48.865983+0000) 2019-10-18T07:36:49.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:49.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:49.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:49.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:49.449 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:49.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:50.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:50.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:50.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:50.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:51.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:51.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:51.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:51.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:52.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:52.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:52.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:52.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:53.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:53.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:53.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:53.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:54.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:54.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:54.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:54.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:54.450 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:54.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:36:55.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:55.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:55.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:55.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:55.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:36:55.399+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:55.405044+0000) 2019-10-18T07:36:56.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:56.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:56.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:56.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:57.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:36:57.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:57.122833+0000) 2019-10-18T07:36:57.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:57.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:57.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:57.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:58.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:58.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:58.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:58.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:58.639 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:36:58.631+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:58.636875+0000) 2019-10-18T07:36:58.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:36:58.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:36:58.866083+0000) 2019-10-18T07:36:59.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:59.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:36:59.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:36:59.075+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:36:59.451 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:36:59.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:00.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:00.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:00.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:00.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:01.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:01.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:01.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:01.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:02.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:02.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:02.063+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:03.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:03.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:03.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:03.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:04.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:04.063+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:04.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:04.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:04.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:04.443+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:05.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:05.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:05.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:05.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:05.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:37:05.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:05.405157+0000) 2019-10-18T07:37:06.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:06.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:06.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:06.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:07.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:37:07.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:07.122946+0000) 2019-10-18T07:37:07.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:07.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:07.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:07.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:08.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:08.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:08.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:08.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:08.639 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:37:08.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:08.637054+0000) 2019-10-18T07:37:08.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:37:08.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:08.866186+0000) 2019-10-18T07:37:09.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:09.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:09.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:09.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:09.452 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:09.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:10.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:10.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:10.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:10.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:11.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:11.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:11.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:11.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:12.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:12.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:12.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:12.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:13.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:13.094+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:13.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:13.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:14.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:14.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:14.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:14.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:14.453 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:14.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:15.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:15.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:15.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:15.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:15.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:37:15.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:15.405266+0000) 2019-10-18T07:37:16.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:16.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:16.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:16.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:17.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:16.994+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:17.005 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:16.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:17.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:37:17.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:17.123057+0000) 2019-10-18T07:37:18.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:18.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:18.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:18.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:18.639 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:37:18.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:18.637262+0000) 2019-10-18T07:37:18.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:37:18.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:18.866289+0000) 2019-10-18T07:37:19.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:19.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:19.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:19.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:19.465 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:19.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:20.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:20.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:20.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:20.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:21.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:21.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:21.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:21.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:22.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:22.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:22.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:22.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:23.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:23.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:23.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:23.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:24.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:24.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:24.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:24.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:24.455 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:24.446+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:25.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:25.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:25.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:25.094+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:25.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:37:25.398+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:25.405373+0000) 2019-10-18T07:37:26.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:26.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:26.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:26.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:27.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:27.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:27.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:27.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:27.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:37:27.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:27.123170+0000) 2019-10-18T07:37:28.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:28.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:28.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:28.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:28.639 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:37:28.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:28.637441+0000) 2019-10-18T07:37:28.868 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:37:28.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:28.866386+0000) 2019-10-18T07:37:29.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:29.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:29.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:29.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:29.456 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:29.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:30.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:30.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:30.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:30.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:31.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:31.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:31.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:31.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:32.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:32.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:32.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:32.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:33.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:33.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:33.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:33.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:34.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:34.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:34.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:34.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:34.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:34.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:35.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:35.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:35.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:35.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:35.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:37:35.402+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:35.405477+0000) 2019-10-18T07:37:36.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:36.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:36.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:36.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:37.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:37.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:37.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:37.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:37.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:37:37.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:37.123284+0000) 2019-10-18T07:37:38.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:38.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:38.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:38.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:38.639 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:37:38.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:38.637619+0000) 2019-10-18T07:37:38.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:37:38.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:38.866488+0000) 2019-10-18T07:37:39.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:39.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:39.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:39.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:39.457 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:39.450+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:40.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:40.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:40.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:40.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:41.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:41.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:41.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:41.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:42.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:42.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:42.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:42.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:43.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:43.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:43.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:43.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:44.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:44.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:44.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:44.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:44.458 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:44.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:45.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:45.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:45.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:45.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:37:45.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:45.405595+0000) 2019-10-18T07:37:46.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:46.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:46.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:46.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:47.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:47.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:47.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:47.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:47.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:37:47.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:47.123397+0000) 2019-10-18T07:37:48.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:48.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:48.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:48.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:48.640 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:37:48.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:48.637807+0000) 2019-10-18T07:37:48.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:37:48.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:48.866588+0000) 2019-10-18T07:37:49.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:49.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:49.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:49.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:49.459 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:49.449+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:50.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:50.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:50.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:50.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:51.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:51.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:51.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:51.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:52.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:52.049+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:52.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:52.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:53.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:53.097+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:53.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:53.097+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:54.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:54.105+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:54.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:54.105+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:54.461 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:54.453+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:37:55.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:55.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:55.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:55.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:55.407 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:37:55.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:55.405701+0000) 2019-10-18T07:37:56.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:56.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:56.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:56.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:57.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:37:57.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:57.123507+0000) 2019-10-18T07:37:57.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:57.153+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:57.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:57.153+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:58.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:58.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:58.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:58.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:58.640 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:37:58.630+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:58.637985+0000) 2019-10-18T07:37:58.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:37:58.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:37:58.866684+0000) 2019-10-18T07:37:59.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:59.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:37:59.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:37:59.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:37:59.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:37:59.457+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:00.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:00.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:00.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:00.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:01.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:01.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:01.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:01.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:02.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:02.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:02.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:02.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:03.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:03.269+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:03.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:03.269+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:04.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:04.241+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:04.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:04.241+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:04.462 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:04.457+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:05.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:05.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:05.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:05.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:05.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:38:05.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:05.405807+0000) 2019-10-18T07:38:06.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:06.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:06.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:06.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:07.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:38:07.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:07.123622+0000) 2019-10-18T07:38:07.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:07.249+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:07.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:07.249+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:08.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:08.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:08.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:08.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:08.640 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:38:08.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:08.638201+0000) 2019-10-18T07:38:08.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:38:08.865+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:08.866786+0000) 2019-10-18T07:38:09.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:09.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:09.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:09.309+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:09.462 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:09.457+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:10.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:10.261+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:10.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:10.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:11.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:11.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:11.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:11.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:12.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:12.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:12.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:12.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:13.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:13.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:13.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:13.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:14.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:14.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:14.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:14.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:14.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:14.461+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:15.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:15.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:15.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:15.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:15.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:38:15.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:15.405912+0000) 2019-10-18T07:38:16.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:16.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:16.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:16.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:17.125 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:38:17.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:17.123731+0000) 2019-10-18T07:38:17.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:17.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:17.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:17.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:18.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:18.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:18.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:18.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:18.640 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:38:18.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:18.638380+0000) 2019-10-18T07:38:18.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:38:18.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:18.866888+0000) 2019-10-18T07:38:19.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:19.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:19.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:19.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:19.464 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:19.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:20.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:20.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:20.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:20.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:21.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:21.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:21.305 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:21.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:22.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:22.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:22.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:22.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:23.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:23.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:23.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:23.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:24.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:24.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:24.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:24.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:24.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:24.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:25.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:25.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:25.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:25.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:25.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:38:25.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:25.406013+0000) 2019-10-18T07:38:26.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:26.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:26.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:26.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:27.126 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:38:27.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:27.123846+0000) 2019-10-18T07:38:27.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:27.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:27.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:27.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:28.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:28.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:28.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:28.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:28.640 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:38:28.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:28.638558+0000) 2019-10-18T07:38:28.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:38:28.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:28.866993+0000) 2019-10-18T07:38:29.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:29.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:29.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:29.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:29.466 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:29.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:30.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:30.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:30.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:30.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:31.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:31.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:31.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:31.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:32.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:32.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:32.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:32.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:33.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:33.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:33.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:33.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:34.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:34.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:34.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:34.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:34.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:34.460+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:35.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:35.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:35.234 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:35.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:35.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:38:35.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:35.406117+0000) 2019-10-18T07:38:36.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:36.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:36.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:36.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:37.126 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:38:37.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:37.123961+0000) 2019-10-18T07:38:37.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:37.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:37.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:37.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:38.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:38.176+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:38.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:38.176+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:38.640 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:38:38.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:38.638722+0000) 2019-10-18T07:38:38.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:38:38.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:38.867094+0000) 2019-10-18T07:38:39.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:39.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:39.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:39.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:39.468 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:39.464+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:40.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:40.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:40.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:40.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:41.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:41.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:41.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:41.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:42.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:42.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:42.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:42.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:43.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:43.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:43.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:43.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:44.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:44.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:44.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:44.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:44.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:44.464+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:45.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:45.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:45.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:45.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:45.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:38:45.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:45.406223+0000) 2019-10-18T07:38:46.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:46.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:46.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:46.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:47.126 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:38:47.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:47.124071+0000) 2019-10-18T07:38:47.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:47.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:47.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:47.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:48.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:48.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:48.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:48.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:48.641 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:38:48.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:48.638900+0000) 2019-10-18T07:38:48.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:38:48.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:48.867193+0000) 2019-10-18T07:38:49.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:49.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:49.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:49.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:49.470 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:49.464+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:50.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:50.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:50.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:50.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:51.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:51.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:51.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:51.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:52.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:52.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:52.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:52.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:53.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:53.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:53.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:53.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:54.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:54.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:54.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:54.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:54.471 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:54.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:38:55.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:55.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:55.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:55.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:55.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:38:55.403+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:55.406279+0000) 2019-10-18T07:38:56.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:56.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:56.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:56.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:57.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:38:57.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:57.124182+0000) 2019-10-18T07:38:57.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:57.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:57.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:57.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:58.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:58.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:58.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:58.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:58.641 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:38:58.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:58.639108+0000) 2019-10-18T07:38:58.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:38:58.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:38:58.867292+0000) 2019-10-18T07:38:59.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:59.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:38:59.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:38:59.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:38:59.472 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:38:59.463+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:00.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:00.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:00.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:00.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:01.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:01.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:01.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:01.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:02.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:02.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:02.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:02.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:03.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:03.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:03.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:04.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:04.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:04.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:04.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:04.472 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:04.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:05.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:05.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:05.207 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:05.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:05.408 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:39:05.403+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:05.406374+0000) 2019-10-18T07:39:06.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:06.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:06.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:06.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:07.126 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:39:07.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:07.124302+0000) 2019-10-18T07:39:07.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:07.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:07.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:07.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:08.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:08.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:08.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:08.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:08.641 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:39:08.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:08.639288+0000) 2019-10-18T07:39:08.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:39:08.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:08.867390+0000) 2019-10-18T07:39:09.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:09.199+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:09.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:09.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:09.473 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:09.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:10.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:10.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:10.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:10.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:11.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:11.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:11.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:11.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:12.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:12.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:12.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:12.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:13.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:13.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:13.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:13.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:14.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:14.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:14.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:14.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:14.475 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:14.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:15.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:15.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:15.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:15.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:15.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:39:15.403+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:15.406477+0000) 2019-10-18T07:39:16.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:16.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:16.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:16.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:17.126 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:39:17.121+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:17.124413+0000) 2019-10-18T07:39:17.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:17.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:17.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:17.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:18.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:18.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:18.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:18.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:18.641 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:39:18.637+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:18.639475+0000) 2019-10-18T07:39:18.869 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:39:18.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:18.867493+0000) 2019-10-18T07:39:19.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:19.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:19.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:19.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:19.476 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:19.467+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:20.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:20.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:20.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:20.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:21.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:21.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:21.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:21.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:22.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:22.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:22.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:22.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:23.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:23.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:23.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:23.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:24.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:24.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:24.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:24.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:24.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:24.471+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:25.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:25.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:25.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:25.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:25.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:39:25.402+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:25.406578+0000) 2019-10-18T07:39:26.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:26.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:26.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:26.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:27.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:27.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:27.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:27.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:39:27.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:27.124521+0000) 2019-10-18T07:39:28.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:28.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:28.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:28.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:28.641 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:39:28.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:28.639687+0000) 2019-10-18T07:39:28.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:39:28.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:28.867592+0000) 2019-10-18T07:39:29.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:29.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:29.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:29.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:29.477 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:29.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:30.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:30.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:30.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:30.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:31.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:31.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:31.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:31.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:32.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:32.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:32.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:32.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:33.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:33.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:33.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:33.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:34.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:34.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:34.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:34.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:34.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:34.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:35.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:35.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:35.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:35.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:35.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:39:35.402+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:35.406684+0000) 2019-10-18T07:39:35.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:35.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:35.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:35.974+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:36.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:36.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:36.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:36.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:37.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:39:37.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:37.124636+0000) 2019-10-18T07:39:37.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:37.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:37.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:37.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:38.642 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:39:38.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:38.639877+0000) 2019-10-18T07:39:38.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:39:38.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:38.867692+0000) 2019-10-18T07:39:38.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:38.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:38.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:38.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:39.478 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:39.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:40.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:39.998+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:40.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:39.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:40.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:40.974+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:40.985 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:40.978+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:41.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:41.938+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:41.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:41.938+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:42.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:42.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:42.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:42.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:43.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:43.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:43.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:43.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:44.479 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:44.470+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:44.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:44.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:44.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:44.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:45.410 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:39:45.402+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:45.406790+0000) 2019-10-18T07:39:45.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:45.822+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:45.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:45.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:46.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:46.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:46.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:46.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:47.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:39:47.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:47.124747+0000) 2019-10-18T07:39:47.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:47.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:47.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:47.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:48.642 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:39:48.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:48.640062+0000) 2019-10-18T07:39:48.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:48.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:48.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:48.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:48.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:39:48.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:48.867792+0000) 2019-10-18T07:39:49.481 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:49.474+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:49.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:49.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:49.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:49.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:50.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:50.798+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:50.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:50.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:51.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:51.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:51.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:51.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:52.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:52.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:52.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:52.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:53.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:53.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:53.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:53.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:54.481 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:54.474+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:54.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:54.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:54.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:54.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:55.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:39:55.402+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:55.406895+0000) 2019-10-18T07:39:55.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:55.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:55.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:55.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:56.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:56.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:56.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:56.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:57.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:39:57.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:57.124858+0000) 2019-10-18T07:39:57.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:57.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:57.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:57.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:58.642 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:39:58.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:58.640270+0000) 2019-10-18T07:39:58.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:58.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:58.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:58.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:39:58.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:39:58.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:39:58.867891+0000) 2019-10-18T07:39:59.482 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:39:59.474+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:39:59.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:59.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:39:59.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:39:59.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:00.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:00.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:00.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:00.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:01.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:01.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:01.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:01.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:02.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:02.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:02.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:02.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:03.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:03.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:03.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:03.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:04.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:04.473+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:04.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:04.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:04.838 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:04.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:05.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:40:05.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:05.407004+0000) 2019-10-18T07:40:05.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:05.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:05.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:05.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:06.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:06.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:06.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:06.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:07.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:40:07.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:07.124975+0000) 2019-10-18T07:40:07.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:07.797+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:07.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:07.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:08.642 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:40:08.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:08.640480+0000) 2019-10-18T07:40:08.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:08.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:08.767 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:08.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:08.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:40:08.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:08.867991+0000) 2019-10-18T07:40:09.483 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:09.477+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:09.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:09.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:09.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:09.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:10.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:10.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:10.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:10.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:11.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:11.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:11.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:11.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:12.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:12.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:12.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:12.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:13.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:13.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:13.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:13.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:14.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:14.477+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:14.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:14.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:14.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:14.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:15.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:40:15.401+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:15.407106+0000) 2019-10-18T07:40:15.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:15.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:15.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:15.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:16.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:16.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:16.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:16.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:17.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:40:17.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:17.125088+0000) 2019-10-18T07:40:17.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:17.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:17.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:17.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:18.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:40:18.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:18.640620+0000) 2019-10-18T07:40:18.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:40:18.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:18.868090+0000) 2019-10-18T07:40:18.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:18.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:18.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:18.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:19.485 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:19.481+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:19.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:19.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:19.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:19.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:20.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:20.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:20.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:20.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:21.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:21.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:21.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:21.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:22.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:22.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:22.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:22.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:23.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:23.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:23.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:23.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:24.487 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:24.481+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:24.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:24.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:24.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:24.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:25.410 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:40:25.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:25.407208+0000) 2019-10-18T07:40:25.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:25.965+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:25.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:25.965+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:26.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:26.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:26.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:26.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:27.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:40:27.120+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:27.125205+0000) 2019-10-18T07:40:27.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:27.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:27.983 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:27.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:28.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:40:28.636+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:28.640853+0000) 2019-10-18T07:40:28.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:40:28.865+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:28.868187+0000) 2019-10-18T07:40:28.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:28.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:28.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:28.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:29.488 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:29.481+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:29.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:29.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:29.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:29.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:30.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:30.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:30.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:30.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:31.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:31.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:31.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:31.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:32.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:32.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:32.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:32.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:33.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:33.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:33.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:33.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:34.488 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:34.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:34.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:34.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:34.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:34.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:35.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:40:35.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:35.407309+0000) 2019-10-18T07:40:35.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:35.960+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:35.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:35.960+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:36.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:36.948+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:36.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:36.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:37.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:40:37.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:37.125319+0000) 2019-10-18T07:40:37.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:37.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:37.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:37.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:38.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:40:38.635+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:38.641035+0000) 2019-10-18T07:40:38.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:40:38.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:38.868286+0000) 2019-10-18T07:40:38.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:38.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:38.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:38.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:39.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:39.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:40.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:39.992+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:40.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:39.996+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:41.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:41.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:41.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:41.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:42.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:42.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:42.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:42.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:43.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:43.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:43.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:43.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:44.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:44.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:44.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:44.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:44.490 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:44.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:45.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:45.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:45.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:45.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:45.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:40:45.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:45.407409+0000) 2019-10-18T07:40:46.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:46.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:46.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:46.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:47.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:47.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:47.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:47.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:47.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:40:47.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:47.125431+0000) 2019-10-18T07:40:48.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:48.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:48.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:48.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:48.643 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:40:48.635+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:48.641225+0000) 2019-10-18T07:40:48.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:40:48.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:48.868385+0000) 2019-10-18T07:40:49.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:49.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:49.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:49.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:49.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:49.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:50.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:50.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:50.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:50.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:51.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:51.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:51.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:51.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:52.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:52.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:52.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:52.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:53.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:53.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:53.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:53.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:54.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:54.168+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:54.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:54.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:54.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:54.484+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:40:55.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:55.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:55.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:55.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:55.409 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:40:55.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:55.407507+0000) 2019-10-18T07:40:56.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:56.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:56.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:56.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:57.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:40:57.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:57.125546+0000) 2019-10-18T07:40:57.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:57.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:57.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:57.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:58.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:58.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:58.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:58.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:58.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:40:58.635+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:58.641403+0000) 2019-10-18T07:40:58.870 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:40:58.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:40:58.868483+0000) 2019-10-18T07:40:59.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:59.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:40:59.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:40:59.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:40:59.493 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:40:59.488+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:00.235 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:00.228+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:00.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:00.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:01.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:01.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:01.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:01.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:02.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:02.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:02.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:02.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:03.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:03.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:03.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:03.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:04.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:04.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:04.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:04.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:04.494 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:04.488+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:05.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:05.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:05.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:05.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:05.410 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:05.404+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:05.407609+0000) 2019-10-18T07:41:05.411 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:05.404+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:05.408719+0000) 2019-10-18T07:41:06.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:06.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:06.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:06.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:07.127 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:41:07.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:07.125650+0000) 2019-10-18T07:41:07.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:07.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:07.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:07.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:08.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:08.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:08.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:08.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:08.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:41:08.635+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:08.641584+0000) 2019-10-18T07:41:08.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:41:08.864+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:08.868588+0000) 2019-10-18T07:41:09.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:09.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:09.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:09.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:09.495 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:09.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:09.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:09.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:09.907814+0000) 2019-10-18T07:41:10.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:10.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:10.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:10.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:11.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:11.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:11.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:11.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:12.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:12.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:12.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:12.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:13.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:13.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:13.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:13.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:14.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:14.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:14.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:14.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:14.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:14.487+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:15.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:15.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:15.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:15.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:16.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:16.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:16.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:16.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:17.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:41:17.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:17.125760+0000) 2019-10-18T07:41:17.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:17.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:17.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:17.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:18.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:18.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:18.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:18.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:18.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:41:18.635+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:18.641763+0000) 2019-10-18T07:41:18.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:41:18.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:18.868690+0000) 2019-10-18T07:41:19.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:19.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:19.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:19.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:19.496 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:19.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:19.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:19.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:19.907915+0000) 2019-10-18T07:41:20.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:20.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:20.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:20.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:21.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:21.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:21.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:21.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:22.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:22.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:22.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:22.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:23.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:23.359+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:23.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:23.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:24.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:24.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:24.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:24.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:24.497 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:24.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:25.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:25.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:25.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:25.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:26.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:26.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:26.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:26.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:27.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:41:27.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:27.125870+0000) 2019-10-18T07:41:27.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:27.431+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:27.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:27.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:28.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:28.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:28.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:28.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:28.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:41:28.635+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:28.641974+0000) 2019-10-18T07:41:28.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:41:28.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:28.868791+0000) 2019-10-18T07:41:29.498 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:29.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:29.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:29.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:29.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:29.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:29.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:29.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:29.908020+0000) 2019-10-18T07:41:30.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:30.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:30.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:30.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:31.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:31.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:31.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:31.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:32.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:32.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:32.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:32.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:33.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:33.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:33.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:33.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:34.499 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:34.491+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:34.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:34.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:34.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:34.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:35.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:35.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:35.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:35.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:36.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:36.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:36.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:36.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:37.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:41:37.119+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:37.125984+0000) 2019-10-18T07:41:37.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:37.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:37.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:37.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:38.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:38.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:38.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:38.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:38.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:41:38.639+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:38.642209+0000) 2019-10-18T07:41:38.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:41:38.863+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:38.868896+0000) 2019-10-18T07:41:39.500 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:39.495+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:39.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:39.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:39.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:39.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:39.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:39.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:39.908120+0000) 2019-10-18T07:41:40.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:40.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:40.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:40.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:41.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:41.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:41.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:41.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:42.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:42.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:43.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:43.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:43.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:43.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:44.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:44.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:44.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:44.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:44.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:44.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:45.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:45.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:45.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:45.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:46.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:46.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:46.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:46.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:47.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:41:47.118+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:47.126096+0000) 2019-10-18T07:41:47.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:47.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:47.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:47.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:48.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:48.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:48.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:48.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:48.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:41:48.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:48.642381+0000) 2019-10-18T07:41:48.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:41:48.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:48.869006+0000) 2019-10-18T07:41:49.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:49.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:49.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:49.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:49.501 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:49.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:49.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:49.902+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:49.908218+0000) 2019-10-18T07:41:50.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:50.490+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:50.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:50.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:51.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:51.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:51.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:51.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:52.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:52.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:52.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:52.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:53.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:53.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:53.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:53.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:54.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:54.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:54.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:54.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:54.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:54.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:55.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:55.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:55.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:55.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:56.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:56.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:56.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:56.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:57.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:41:57.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:57.126213+0000) 2019-10-18T07:41:57.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:57.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:57.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:57.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:58.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:58.542+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:58.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:58.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:58.644 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:41:58.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:58.642528+0000) 2019-10-18T07:41:58.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:41:58.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:58.869105+0000) 2019-10-18T07:41:59.503 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:41:59.494+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:41:59.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:59.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:41:59.555 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:41:59.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:41:59.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:41:59.902+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:41:59.908322+0000) 2019-10-18T07:42:00.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:00.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:00.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:00.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:01.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:01.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:01.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:01.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:02.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:02.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:02.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:02.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:03.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:03.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:03.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:03.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:04.504 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:04.498+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:04.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:04.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:04.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:04.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:05.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:05.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:05.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:05.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:06.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:06.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:06.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:06.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:07.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:42:07.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:07.126312+0000) 2019-10-18T07:42:07.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:07.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:07.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:07.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:08.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:08.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:08.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:08.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:08.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:42:08.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:08.642716+0000) 2019-10-18T07:42:08.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:42:08.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:08.869208+0000) 2019-10-18T07:42:09.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:09.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:09.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:09.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:09.505 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:09.498+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:09.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:42:09.902+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:09.908424+0000) 2019-10-18T07:42:10.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:10.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:10.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:10.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:11.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:11.546+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:11.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:11.546+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:12.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:12.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:12.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:12.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:13.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:13.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:13.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:13.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:14.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:14.498+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:14.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:14.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:14.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:14.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:15.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:15.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:15.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:15.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:16.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:16.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:16.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:16.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:17.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:42:17.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:17.126412+0000) 2019-10-18T07:42:17.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:17.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:17.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:17.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:18.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:18.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:18.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:18.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:18.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:42:18.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:18.642898+0000) 2019-10-18T07:42:18.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:42:18.862+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:18.869308+0000) 2019-10-18T07:42:19.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:19.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:19.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:19.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:19.507 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:19.497+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:19.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:42:19.901+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:19.908520+0000) 2019-10-18T07:42:20.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:20.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:20.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:20.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:21.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:21.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:21.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:21.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:22.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:22.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:22.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:22.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:23.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:23.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:23.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:23.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:24.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:24.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:24.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:24.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:24.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:24.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:25.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:25.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:25.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:25.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:26.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:26.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:26.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:26.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:27.128 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:42:27.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:27.126506+0000) 2019-10-18T07:42:27.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:27.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:27.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:27.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:28.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:28.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:28.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:28.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:28.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:42:28.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:28.643111+0000) 2019-10-18T07:42:28.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:42:28.861+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:28.869406+0000) 2019-10-18T07:42:29.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:29.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:29.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:29.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:29.509 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:29.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:29.910 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:42:29.901+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:29.908617+0000) 2019-10-18T07:42:30.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:30.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:30.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:30.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:31.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:31.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:31.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:31.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:32.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:32.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:32.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:32.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:33.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:33.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:33.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:33.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:34.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:34.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:34.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:34.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:34.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:34.501+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:35.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:35.341+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:35.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:35.341+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:36.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:36.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:36.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:36.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:37.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:42:37.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:37.126619+0000) 2019-10-18T07:42:37.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:37.285+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:37.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:37.285+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:38.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:38.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:38.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:38.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:38.645 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:42:38.638+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:38.643290+0000) 2019-10-18T07:42:38.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:42:38.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:38.869501+0000) 2019-10-18T07:42:39.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:39.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:39.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:39.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:39.510 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:39.505+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:39.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:42:39.905+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:39.908726+0000) 2019-10-18T07:42:40.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:40.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:40.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:40.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:41.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:41.245+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:41.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:41.245+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:42.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:42.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:42.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:42.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:43.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:43.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:43.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:43.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:44.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:44.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:44.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:44.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:44.511 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:44.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:45.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:45.309+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:45.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:45.313+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:46.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:46.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:46.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:46.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:47.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:42:47.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:47.126732+0000) 2019-10-18T07:42:47.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:47.277+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:47.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:47.277+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:48.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:48.281+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:48.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:48.281+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:48.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:42:48.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:48.643499+0000) 2019-10-18T07:42:48.871 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:42:48.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:48.869598+0000) 2019-10-18T07:42:49.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:49.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:49.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:49.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:49.513 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:49.509+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:49.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:42:49.905+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:49.908822+0000) 2019-10-18T07:42:50.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:50.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:50.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:50.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:51.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:51.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:51.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:51.257+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:52.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:52.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:52.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:52.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:53.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:53.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:53.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:53.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:54.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:54.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:54.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:54.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:54.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:54.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:55.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:55.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:55.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:55.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:56.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:56.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:56.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:56.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:57.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:42:57.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:57.126846+0000) 2019-10-18T07:42:57.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:57.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:57.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:57.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:58.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:58.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:58.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:58.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:58.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:42:58.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:58.643711+0000) 2019-10-18T07:42:58.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:42:58.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:58.869691+0000) 2019-10-18T07:42:59.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:59.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:42:59.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:42:59.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:42:59.514 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:42:59.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:42:59.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:42:59.904+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:42:59.908926+0000) 2019-10-18T07:43:00.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:00.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:00.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:00.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:01.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:01.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:01.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:01.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:02.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:02.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:02.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:02.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:03.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:03.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:03.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:03.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:04.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:04.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:04.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:04.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:04.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:04.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:05.281 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:05.276+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:05.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:05.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:06.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:06.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:06.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:06.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:07.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:43:07.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:07.126960+0000) 2019-10-18T07:43:07.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:07.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:07.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:07.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:08.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:08.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:08.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:08.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:08.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:43:08.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:08.643885+0000) 2019-10-18T07:43:08.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:43:08.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:08.869793+0000) 2019-10-18T07:43:09.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:09.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:09.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:09.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:09.515 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:09.508+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:09.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:09.904+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:09.909027+0000) 2019-10-18T07:43:10.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:10.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:10.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:10.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:11.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:11.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:11.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:11.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:12.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:12.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:12.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:12.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:13.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:13.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:13.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:13.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:14.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:14.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:14.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:14.292+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:14.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:14.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:15.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:15.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:15.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:15.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:16.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:16.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:16.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:16.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:17.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:43:17.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:17.127065+0000) 2019-10-18T07:43:17.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:17.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:17.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:17.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:18.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:18.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:18.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:18.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:18.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:43:18.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:18.644038+0000) 2019-10-18T07:43:18.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:43:18.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:18.869886+0000) 2019-10-18T07:43:19.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:19.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:19.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:19.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:19.518 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:19.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:19.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:19.904+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:19.909124+0000) 2019-10-18T07:43:20.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:20.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:20.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:20.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:21.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:21.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:21.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:21.412+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:22.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:22.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:22.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:22.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:23.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:23.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:23.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:23.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:24.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:24.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:24.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:24.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:24.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:24.512+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:25.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:25.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:25.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:25.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:26.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:26.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:26.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:26.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:27.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:43:27.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:27.127159+0000) 2019-10-18T07:43:27.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:27.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:27.486 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:27.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:28.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:28.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:28.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:28.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:28.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:43:28.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:28.644233+0000) 2019-10-18T07:43:28.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:43:28.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:28.869985+0000) 2019-10-18T07:43:29.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:29.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:29.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:29.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:29.519 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:29.511+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:29.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:29.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:29.909230+0000) 2019-10-18T07:43:30.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:30.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:30.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:30.439+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:31.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:31.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:31.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:31.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:32.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:32.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:32.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:32.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:33.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:33.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:33.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:33.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:34.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:34.435+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:34.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:34.435+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:34.520 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:34.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:35.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:35.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:35.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:35.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:36.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:36.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:36.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:36.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:37.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:43:37.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:37.127271+0000) 2019-10-18T07:43:37.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:37.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:37.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:37.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:38.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:38.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:38.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:38.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:38.646 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:43:38.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:38.644443+0000) 2019-10-18T07:43:38.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:43:38.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:38.870088+0000) 2019-10-18T07:43:39.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:39.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:39.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:39.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:39.521 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:39.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:39.911 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:39.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:39.909329+0000) 2019-10-18T07:43:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:40.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:40.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:41.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:41.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:41.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:41.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:42.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:42.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:42.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:42.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:43.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:43.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:43.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:43.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:44.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:44.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:44.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:44.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:44.522 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:44.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:45.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:45.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:45.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:45.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:46.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:46.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:46.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:46.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:47.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:43:47.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:47.127383+0000) 2019-10-18T07:43:47.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:47.463+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:47.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:47.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:48.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:48.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:48.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:48.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:48.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:43:48.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:48.644623+0000) 2019-10-18T07:43:48.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:43:48.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:48.870185+0000) 2019-10-18T07:43:49.523 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:49.515+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:49.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:49.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:49.912 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:49.903+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:49.909427+0000) 2019-10-18T07:43:50.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:50.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:50.573 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:50.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:51.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:51.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:51.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:51.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:52.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:52.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:52.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:52.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:53.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:53.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:53.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:53.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:54.524 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:54.519+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:54.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:54.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:54.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:54.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:55.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:55.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:55.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:55.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:56.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:56.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:56.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:56.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:57.129 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:43:57.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:57.127495+0000) 2019-10-18T07:43:57.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:57.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:57.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:57.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:58.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:58.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:58.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:58.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:58.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:43:58.641+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:58.644801+0000) 2019-10-18T07:43:58.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:43:58.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:58.870283+0000) 2019-10-18T07:43:59.525 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:43:59.519+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:43:59.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:59.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:43:59.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:43:59.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:43:59.912 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:59.907+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:59.909613+0000) 2019-10-18T07:43:59.913 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:43:59.907+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:43:59.910732+0000) 2019-10-18T07:44:00.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:00.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:00.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:00.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:01.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:01.535+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:01.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:01.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:02.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:02.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:02.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:02.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:03.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:03.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:03.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:03.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:04.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:44:04.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:04.409813+0000) 2019-10-18T07:44:04.526 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:04.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:04.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:04.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:04.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:04.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:05.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:05.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:05.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:05.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:06.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:06.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:06.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:06.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:07.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:44:07.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:07.127616+0000) 2019-10-18T07:44:07.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:07.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:07.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:07.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:08.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:08.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:08.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:08.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:08.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:44:08.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:08.645012+0000) 2019-10-18T07:44:08.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:44:08.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:08.870383+0000) 2019-10-18T07:44:09.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:09.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:09.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:09.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:09.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:09.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:10.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:10.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:10.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:10.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:11.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:11.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:11.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:11.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:12.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:12.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:12.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:12.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:13.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:13.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:13.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:13.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:14.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:44:14.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:14.409914+0000) 2019-10-18T07:44:14.527 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:14.518+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:14.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:14.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:14.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:14.542+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:15.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:15.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:15.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:15.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:16.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:16.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:16.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:16.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:17.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:44:17.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:17.127727+0000) 2019-10-18T07:44:17.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:17.522+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:17.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:17.522+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:18.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:18.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:18.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:18.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:18.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:44:18.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:18.645223+0000) 2019-10-18T07:44:18.872 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:44:18.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:18.870481+0000) 2019-10-18T07:44:19.528 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:19.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:19.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:19.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:19.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:19.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:20.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:20.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:20.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:20.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:21.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:21.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:21.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:21.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:22.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:22.450+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:22.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:22.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:23.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:23.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:23.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:23.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:24.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:24.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:24.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:24.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:24.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:44:24.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:24.410015+0000) 2019-10-18T07:44:24.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:24.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:25.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:25.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:25.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:25.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:26.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:26.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:26.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:26.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:27.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:44:27.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:27.127838+0000) 2019-10-18T07:44:27.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:27.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:27.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:27.450+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:28.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:28.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:28.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:28.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:28.647 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:44:28.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:28.645435+0000) 2019-10-18T07:44:28.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:44:28.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:28.870570+0000) 2019-10-18T07:44:29.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:29.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:29.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:29.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:29.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:29.522+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:30.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:30.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:30.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:30.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:31.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:31.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:31.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:31.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:32.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:32.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:32.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:32.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:33.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:33.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:33.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:33.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:34.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:44:34.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:34.410115+0000) 2019-10-18T07:44:34.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:34.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:34.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:34.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:34.532 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:34.526+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:35.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:35.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:35.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:35.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:36.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:36.406+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:36.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:36.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:37.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:44:37.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:37.127951+0000) 2019-10-18T07:44:37.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:37.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:37.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:37.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:38.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:38.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:38.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:38.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:38.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:44:38.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:38.645615+0000) 2019-10-18T07:44:38.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:44:38.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:38.870669+0000) 2019-10-18T07:44:39.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:39.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:39.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:39.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:39.533 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:39.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:40.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:40.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:40.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:40.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:41.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:41.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:41.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:41.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:42.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:42.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:42.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:42.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:43.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:43.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:43.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:43.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:44.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:44.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:44.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:44.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:44.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:44:44.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:44.410219+0000) 2019-10-18T07:44:44.534 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:44.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:45.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:45.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:45.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:45.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:46.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:46.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:46.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:46.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:47.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:44:47.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:47.128063+0000) 2019-10-18T07:44:47.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:47.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:47.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:47.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:48.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:48.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:48.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:48.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:48.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:44:48.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:48.645795+0000) 2019-10-18T07:44:48.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:44:48.865+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:48.870768+0000) 2019-10-18T07:44:49.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:49.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:49.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:49.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:49.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:49.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:50.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:50.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:50.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:50.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:51.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:51.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:51.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:51.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:52.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:52.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:52.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:52.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:53.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:53.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:53.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:53.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:54.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:44:54.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:54.410315+0000) 2019-10-18T07:44:54.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:54.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:54.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:54.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:54.535 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:54.525+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:44:55.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:55.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:55.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:55.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:56.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:56.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:56.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:56.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:57.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:44:57.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:57.128174+0000) 2019-10-18T07:44:57.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:57.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:57.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:57.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:58.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:58.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:58.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:58.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:58.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:44:58.640+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:58.645980+0000) 2019-10-18T07:44:58.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:44:58.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:44:58.870865+0000) 2019-10-18T07:44:59.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:59.345+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:44:59.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:44:59.345+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:44:59.536 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:44:59.529+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:00.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:00.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:00.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:00.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:01.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:01.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:01.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:01.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:02.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:02.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:02.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:02.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:03.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:03.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:03.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:03.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:04.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:04.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:04.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:04.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:04.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:45:04.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:04.410419+0000) 2019-10-18T07:45:04.537 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:04.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:05.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:05.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:05.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:05.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:06.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:06.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:06.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:06.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:07.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:45:07.124+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:07.128287+0000) 2019-10-18T07:45:07.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:07.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:07.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:07.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:08.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:08.329+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:08.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:08.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:08.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:45:08.644+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:08.646239+0000) 2019-10-18T07:45:08.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:45:08.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:08.870965+0000) 2019-10-18T07:45:09.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:09.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:09.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:09.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:09.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:09.533+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:10.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:10.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:10.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:10.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:11.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:11.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:11.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:11.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:12.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:12.297+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:12.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:12.297+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:13.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:13.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:13.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:13.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:14.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:14.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:14.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:14.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:14.412 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:45:14.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:14.410523+0000) 2019-10-18T07:45:14.538 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:14.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:15.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:15.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:15.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:15.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:16.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:16.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:16.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:16.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:17.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:45:17.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:17.128399+0000) 2019-10-18T07:45:17.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:17.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:17.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:17.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:18.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:18.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:18.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:18.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:18.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:45:18.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:18.646385+0000) 2019-10-18T07:45:18.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:45:18.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:18.871063+0000) 2019-10-18T07:45:19.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:19.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:19.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:19.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:19.539 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:19.532+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:20.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:20.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:20.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:20.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:21.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:21.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:21.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:21.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:22.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:22.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:22.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:22.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:23.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:23.292+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:23.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:23.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:24.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:24.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:24.325 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:24.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:24.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:45:24.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:24.410625+0000) 2019-10-18T07:45:24.540 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:24.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:25.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:25.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:25.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:25.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:26.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:26.336+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:26.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:26.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:27.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:45:27.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:27.128512+0000) 2019-10-18T07:45:27.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:27.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:27.341 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:27.336+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:28.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:28.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:28.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:28.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:28.648 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:45:28.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:28.646553+0000) 2019-10-18T07:45:28.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:45:28.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:28.871165+0000) 2019-10-18T07:45:29.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:29.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:29.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:29.340+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:29.541 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:29.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:30.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:30.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:30.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:30.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:31.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:31.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:31.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:31.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:32.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:32.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:32.333 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:32.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:33.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:33.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:33.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:33.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:34.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:34.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:34.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:34.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:34.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:45:34.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:34.410727+0000) 2019-10-18T07:45:34.542 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:34.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:35.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:35.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:35.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:35.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:36.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:36.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:36.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:36.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:37.130 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:45:37.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:37.128625+0000) 2019-10-18T07:45:37.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:37.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:37.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:37.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:38.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:38.312+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:38.318 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:38.312+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:38.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:45:38.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:38.646742+0000) 2019-10-18T07:45:38.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:45:38.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:38.871266+0000) 2019-10-18T07:45:39.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:39.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:39.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:39.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:39.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:39.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:40.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:40.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:40.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:40.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:41.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:41.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:41.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:41.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:42.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:42.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:42.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:42.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:43.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:43.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:43.285 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:43.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:44.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:44.296+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:44.302 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:44.296+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:44.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:45:44.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:44.410829+0000) 2019-10-18T07:45:44.543 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:44.536+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:45.316+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:45.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:45.316+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:46.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:46.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:46.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:46.328+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:47.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:45:47.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:47.128744+0000) 2019-10-18T07:45:47.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:47.324+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:47.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:47.324+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:48.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:48.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:48.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:48.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:48.649 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:45:48.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:48.646921+0000) 2019-10-18T07:45:48.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:45:48.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:48.871370+0000) 2019-10-18T07:45:49.327 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:49.319+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:49.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:49.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:49.544 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:49.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:50.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:50.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:50.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:50.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:51.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:51.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:51.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:51.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:52.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:52.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:52.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:52.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:53.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:53.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:53.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:53.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:54.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:54.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:54.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:54.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:54.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:45:54.407+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:54.410930+0000) 2019-10-18T07:45:54.545 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:54.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:45:55.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:55.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:55.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:55.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:56.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:56.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:56.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:56.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:57.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:45:57.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:57.128855+0000) 2019-10-18T07:45:57.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:57.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:57.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:57.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:58.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:58.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:58.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:58.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:58.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:45:58.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:58.647106+0000) 2019-10-18T07:45:58.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:45:58.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:45:58.871475+0000) 2019-10-18T07:45:59.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:59.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:45:59.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:45:59.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:45:59.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:45:59.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:00.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:00.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:00.182 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:00.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:01.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:01.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:01.161 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:01.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:02.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:02.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:02.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:02.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:03.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:03.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:03.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:03.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:04.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:04.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:04.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:04.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:04.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:46:04.407+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:04.411033+0000) 2019-10-18T07:46:04.547 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:04.539+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:05.178 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:05.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:05.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:05.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:06.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:06.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:06.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:06.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:07.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:46:07.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:07.128968+0000) 2019-10-18T07:46:07.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:07.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:07.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:07.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:08.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:08.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:08.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:08.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:08.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:46:08.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:08.647298+0000) 2019-10-18T07:46:08.873 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:46:08.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:08.871572+0000) 2019-10-18T07:46:09.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:09.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:09.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:09.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:09.548 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:09.543+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:10.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:10.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:10.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:10.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:11.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:11.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:11.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:11.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:12.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:12.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:12.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:12.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:13.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:13.151+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:13.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:13.151+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:14.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:14.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:14.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:14.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:14.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:46:14.407+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:14.411134+0000) 2019-10-18T07:46:14.549 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:14.543+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:15.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:15.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:15.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:15.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:16.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:16.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:16.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:16.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:17.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:46:17.123+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:17.129079+0000) 2019-10-18T07:46:17.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:17.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:17.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:17.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:18.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:18.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:18.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:18.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:18.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:46:18.643+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:18.647488+0000) 2019-10-18T07:46:18.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:46:18.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:18.871672+0000) 2019-10-18T07:46:19.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:19.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:19.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:19.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:19.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:19.543+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:20.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:20.167 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:20.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:21.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:21.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:21.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:21.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:22.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:22.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:22.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:22.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:23.163 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:23.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:23.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:23.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:24.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:24.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:24.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:24.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:24.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:46:24.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:24.411238+0000) 2019-10-18T07:46:24.550 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:24.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:25.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:25.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:25.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:25.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:26.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:26.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:26.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:26.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:27.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:27.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:27.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:27.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:27.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:46:27.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:27.129190+0000) 2019-10-18T07:46:28.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:28.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:28.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:28.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:28.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:46:28.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:28.647677+0000) 2019-10-18T07:46:28.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:46:28.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:28.871767+0000) 2019-10-18T07:46:29.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:29.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:29.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:29.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:29.551 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:29.542+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:30.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:30.090+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:30.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:30.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:31.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:31.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:31.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:31.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:32.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:32.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:32.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:32.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:33.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:33.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:33.022 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:33.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:33.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:33.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:33.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:33.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:34.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:46:34.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:34.411334+0000) 2019-10-18T07:46:34.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:34.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:35.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:34.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:35.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:34.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:35.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:36:05.658781+0000 (oldest deadline 2019-10-18T07:36:25.658781+0000) 2019-10-18T07:46:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:35.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:36.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:36.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:37.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:46:37.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:37.129298+0000) 2019-10-18T07:46:37.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:37.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:38.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:46:38.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:38.647838+0000) 2019-10-18T07:46:38.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:46:38.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:38.871869+0000) 2019-10-18T07:46:38.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:38.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:39.553 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:39.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:39.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:39.886+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:40.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:40.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:41.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:41.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:42.945 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:42.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:43.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:43.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:44.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:46:44.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:44.411429+0000) 2019-10-18T07:46:44.554 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:44.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:44.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:44.922+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:45.909 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:45.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:46.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:46.882+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:47.131 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:46:47.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:47.129412+0000) 2019-10-18T07:46:47.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:47.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:48.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:46:48.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:48.648017+0000) 2019-10-18T07:46:48.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:48.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:48.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:46:48.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:48.871970+0000) 2019-10-18T07:46:49.555 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:49.546+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:49.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:49.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:50.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:50.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:51.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:51.918+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:52.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:52.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:53.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:53.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:54.413 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:46:54.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:54.411531+0000) 2019-10-18T07:46:54.556 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:54.550+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:46:54.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:54.930+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:55.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:55.982+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:57.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:56.998+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:57.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:46:57.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:57.129529+0000) 2019-10-18T07:46:57.966 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:57.958+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:58.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:46:58.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:58.648222+0000) 2019-10-18T07:46:58.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:46:58.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:46:58.872075+0000) 2019-10-18T07:46:58.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:46:58.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:46:59.556 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:46:59.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:00.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:00.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:00.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:00.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:00.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:00.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:01.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:01.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:01.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:01.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:02.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:02.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:02.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:02.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:03.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:03.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:03.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:03.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:04.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:47:04.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:04.411630+0000) 2019-10-18T07:47:04.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:04.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:04.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:04.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:04.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:04.861+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:05.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:05.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:05.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:05.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:06.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:06.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:06.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:06.773+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:07.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:47:07.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:07.129640+0000) 2019-10-18T07:47:07.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:07.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:07.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:07.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:08.650 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:47:08.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:08.648429+0000) 2019-10-18T07:47:08.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:08.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:08.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:08.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:08.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:47:08.865+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:08.872176+0000) 2019-10-18T07:47:09.558 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:09.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:09.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:09.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:09.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:10.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:10.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:10.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:10.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:11.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:11.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:11.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:11.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:12.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:12.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:12.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:12.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:13.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:13.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:13.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:13.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:14.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:47:14.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:14.411732+0000) 2019-10-18T07:47:14.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:14.549+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:14.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:14.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:14.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:14.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:15.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:15.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:15.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:15.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:16.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:16.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:16.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:16.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:17.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:47:17.122+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:17.129753+0000) 2019-10-18T07:47:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:17.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:17.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:17.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:18.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:47:18.642+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:18.648629+0000) 2019-10-18T07:47:18.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:18.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:18.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:18.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:18.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:47:18.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:18.872277+0000) 2019-10-18T07:47:19.559 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:19.553+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:19.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:19.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:19.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:19.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:20.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:20.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:20.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:20.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:21.880 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:21.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:21.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:21.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:22.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:22.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:22.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:22.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:23.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:23.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:23.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:23.873+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:24.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:47:24.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:24.411833+0000) 2019-10-18T07:47:24.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:24.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:24.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:24.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:24.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:24.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:25.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:25.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:25.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:25.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:26.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:26.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:26.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:26.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:27.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:47:27.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:27.129870+0000) 2019-10-18T07:47:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:27.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:27.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:27.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:28.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:47:28.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:28.648807+0000) 2019-10-18T07:47:28.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:47:28.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:28.872378+0000) 2019-10-18T07:47:28.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:28.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:28.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:28.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:29.561 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:29.557+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:29.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:29.833+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:29.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:29.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:30.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:30.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:30.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:30.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:31.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:31.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:31.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:31.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:32.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:32.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:32.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:32.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:33.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:33.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:33.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:33.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:34.428 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:47:34.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:34.411931+0000) 2019-10-18T07:47:34.562 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:34.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:34.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:34.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:34.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:34.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:35.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:35.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:35.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:35.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:36.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:36.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:36.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:36.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:37.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:47:37.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:37.129985+0000) 2019-10-18T07:47:37.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:37.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:37.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:37.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:38.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:47:38.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:38.648987+0000) 2019-10-18T07:47:38.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:38.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:38.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:38.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:38.874 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:47:38.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:38.872481+0000) 2019-10-18T07:47:39.563 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:39.556+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:39.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:39.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:39.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:39.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:40.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:40.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:40.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:40.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:41.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:41.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:41.798 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:41.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:42.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:42.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:42.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:42.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:43.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:43.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:43.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:43.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:44.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:47:44.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:44.412032+0000) 2019-10-18T07:47:44.564 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:44.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:44.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:44.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:44.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:44.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:45.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:45.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:45.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:45.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:46.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:46.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:46.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:46.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:47.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:47:47.125+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:47.130101+0000) 2019-10-18T07:47:47.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:47.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:47.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:47.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:48.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:47:48.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:48.649167+0000) 2019-10-18T07:47:48.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:48.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:48.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:48.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:48.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:47:48.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:48.872608+0000) 2019-10-18T07:47:49.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:49.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:49.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:49.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:49.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:49.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:50.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:50.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:50.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:50.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:51.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:51.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:51.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:51.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:52.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:52.584+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:52.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:52.584+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:53.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:53.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:53.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:53.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:54.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:47:54.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:54.412131+0000) 2019-10-18T07:47:54.565 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:54.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:54.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:54.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:54.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:54.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:55.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:55.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:55.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:55.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:56.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:56.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:56.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:56.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:57.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:47:57.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:57.130215+0000) 2019-10-18T07:47:57.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:57.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:57.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:57.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:58.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:58.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:58.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:58.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:47:58.651 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:47:58.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:58.649345+0000) 2019-10-18T07:47:58.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:47:58.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:47:58.872720+0000) 2019-10-18T07:47:59.566 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:47:59.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:47:59.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:59.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:47:59.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:47:59.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:00.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:00.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:00.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:00.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:01.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:01.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:01.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:01.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:02.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:02.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:02.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:02.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:03.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:03.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:03.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:03.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:04.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:48:04.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:04.412232+0000) 2019-10-18T07:48:04.567 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:04.560+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:04.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:04.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:04.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:04.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:05.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:05.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:05.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:05.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:06.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:06.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:06.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:06.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:07.137 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:48:07.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:07.130315+0000) 2019-10-18T07:48:07.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:07.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:07.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:07.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:08.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:48:08.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:08.649529+0000) 2019-10-18T07:48:08.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:08.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:08.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:08.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:08.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:48:08.868+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:08.872839+0000) 2019-10-18T07:48:09.568 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:09.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:09.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:09.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:09.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:09.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:10.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:10.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:10.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:10.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:11.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:11.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:11.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:11.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:12.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:12.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:12.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:12.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:13.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:13.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:13.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:13.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:14.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:48:14.407+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:14.412333+0000) 2019-10-18T07:48:14.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:14.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:14.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:14.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:14.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:14.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:15.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:15.587+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:15.594 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:15.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:16.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:16.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:16.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:16.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:17.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:48:17.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:17.130415+0000) 2019-10-18T07:48:17.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:17.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:17.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:17.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:18.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:18.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:18.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:18.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:18.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:48:18.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:18.649725+0000) 2019-10-18T07:48:18.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:48:18.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:18.872960+0000) 2019-10-18T07:48:19.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:19.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:19.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:19.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:19.569 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:19.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:20.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:20.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:20.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:20.531+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:21.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:21.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:21.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:21.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:22.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:22.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:22.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:22.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:23.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:23.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:23.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:23.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:24.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:48:24.407+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:24.412435+0000) 2019-10-18T07:48:24.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:24.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:24.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:24.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:24.570 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:24.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:25.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:25.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:25.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:25.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:26.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:26.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:26.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:26.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:27.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:48:27.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:27.130515+0000) 2019-10-18T07:48:27.600 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:27.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:27.601 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:27.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:28.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:28.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:28.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:28.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:28.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:48:28.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:28.649906+0000) 2019-10-18T07:48:28.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:48:28.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:28.873080+0000) 2019-10-18T07:48:29.571 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:29.563+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:29.611 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:29.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:29.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:29.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:30.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:30.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:30.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:30.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:31.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:31.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:31.596 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:31.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:32.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:32.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:32.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:32.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:33.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:33.507+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:33.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:33.507+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:34.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:48:34.407+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:34.412536+0000) 2019-10-18T07:48:34.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:34.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:34.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:34.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:34.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:34.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:35.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:35.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:35.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:35.515+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:36.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:36.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:36.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:36.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:37.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:48:37.129+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:37.130614+0000) 2019-10-18T07:48:37.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:37.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:37.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:37.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:38.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:38.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:38.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:38.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:38.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:48:38.645+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:38.650087+0000) 2019-10-18T07:48:38.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:48:38.867+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:38.873197+0000) 2019-10-18T07:48:39.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:39.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:39.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:39.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:39.573 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:39.567+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:40.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:40.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:40.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:40.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:41.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:41.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:41.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:41.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:42.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:42.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:42.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:42.391+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:43.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:43.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:43.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:43.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:44.414 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:48:44.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:44.412633+0000) 2019-10-18T07:48:44.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:44.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:44.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:44.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:44.574 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:44.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:45.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:45.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:45.440 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:45.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:46.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:46.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:46.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:46.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:47.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:48:47.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:47.130731+0000) 2019-10-18T07:48:47.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:47.506+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:47.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:47.506+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:48.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:48.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:48.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:48.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:48.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:48:48.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:48.650269+0000) 2019-10-18T07:48:48.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:48:48.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:48.873316+0000) 2019-10-18T07:48:49.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:49.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:49.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:49.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:49.575 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:49.566+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:50.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:50.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:50.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:50.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:51.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:51.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:51.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:51.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:52.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:52.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:52.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:52.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:53.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:53.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:53.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:53.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:54.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:54.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:54.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:54.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:48:54.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:54.412732+0000) 2019-10-18T07:48:54.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:54.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:48:55.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:55.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:55.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:55.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:56.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:56.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:56.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:56.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:57.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:48:57.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:57.130844+0000) 2019-10-18T07:48:57.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:57.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:57.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:57.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:58.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:58.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:58.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:58.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:58.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:48:58.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:58.650411+0000) 2019-10-18T07:48:58.875 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:48:58.866+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:48:58.873430+0000) 2019-10-18T07:48:59.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:59.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:48:59.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:48:59.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:48:59.576 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:48:59.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:00.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:00.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:00.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:00.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:01.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:01.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:01.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:02.404 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:02.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:02.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:02.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:03.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:03.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:03.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:03.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:04.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:04.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:04.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:04.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:04.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:49:04.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:04.412834+0000) 2019-10-18T07:49:04.577 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:04.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:05.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:05.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:05.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:05.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:06.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:06.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:06.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:06.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:07.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:49:07.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:07.130955+0000) 2019-10-18T07:49:07.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:07.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:07.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:07.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:08.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:08.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:08.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:08.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:08.652 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:49:08.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:08.650558+0000) 2019-10-18T07:49:08.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:49:08.870+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:08.873579+0000) 2019-10-18T07:49:09.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:09.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:09.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:09.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:09.579 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:09.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:10.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:10.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:10.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:10.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:11.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:11.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:11.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:11.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:12.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:12.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:12.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:12.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:13.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:13.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:13.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:13.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:14.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:49:14.406+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:14.412935+0000) 2019-10-18T07:49:14.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:14.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:14.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:14.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:14.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:14.570+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:15.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:15.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:15.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:15.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:16.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:16.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:16.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:16.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:17.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:49:17.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:17.131063+0000) 2019-10-18T07:49:17.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:17.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:17.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:17.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:18.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:18.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:18.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:18.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:18.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:49:18.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:18.650726+0000) 2019-10-18T07:49:18.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:49:18.870+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:18.873699+0000) 2019-10-18T07:49:19.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:19.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:19.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:19.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:19.580 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:19.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:20.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:20.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:20.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:21.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:21.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:21.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:21.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:22.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:22.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:22.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:22.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:23.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:23.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:23.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:23.489+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:24.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:49:24.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:24.413038+0000) 2019-10-18T07:49:24.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:24.509+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:24.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:24.509+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:24.581 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:24.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:25.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:25.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:25.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:25.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:26.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:26.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:26.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:26.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:27.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:49:27.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:27.131171+0000) 2019-10-18T07:49:27.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:27.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:27.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:27.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:28.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:28.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:28.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:28.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:28.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:49:28.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:28.650933+0000) 2019-10-18T07:49:28.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:49:28.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:28.873811+0000) 2019-10-18T07:49:29.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:29.485+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:29.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:29.485+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:29.582 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:29.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:30.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:30.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:30.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:30.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:31.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:31.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:31.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:31.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:32.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:32.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:32.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:32.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:33.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:33.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:33.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:33.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:34.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:34.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:34.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:34.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:34.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:49:34.405+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:34.413140+0000) 2019-10-18T07:49:34.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:34.573+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:35.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:35.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:35.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:35.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:36.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:36.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:36.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:36.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:37.133 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:49:37.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:37.131282+0000) 2019-10-18T07:49:37.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:37.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:37.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:37.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:38.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:38.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:38.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:38.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:38.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:49:38.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:38.651141+0000) 2019-10-18T07:49:38.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:49:38.873+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:38.873922+0000) 2019-10-18T07:49:39.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:39.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:39.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:39.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:39.583 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:39.577+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:40.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:40.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:40.377 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:40.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:41.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:41.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:41.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:41.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:42.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:42.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:42.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:42.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:43.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:43.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:43.371 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:43.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:44.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:44.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:44.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:44.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:44.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:49:44.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:44.413242+0000) 2019-10-18T07:49:44.584 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:44.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:45.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:45.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:45.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:45.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:46.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:46.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:46.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:46.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:47.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:49:47.128+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:47.131408+0000) 2019-10-18T07:49:47.446 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:47.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:47.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:47.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:48.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:48.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:48.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:48.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:48.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:49:48.648+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:48.651346+0000) 2019-10-18T07:49:48.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:49:48.873+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:48.874038+0000) 2019-10-18T07:49:49.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:49.501+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:49.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:49.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:49.585 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:49.581+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:50.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:50.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:50.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:50.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:51.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:51.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:51.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:51.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:52.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:52.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:52.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:52.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:53.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:53.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:53.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:53.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:54.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:54.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:54.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:54.384+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:54.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:49:54.408+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:54.413351+0000) 2019-10-18T07:49:54.586 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:54.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:49:55.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:55.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:55.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:55.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:56.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:56.328+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:56.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:56.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:57.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:49:57.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:57.131522+0000) 2019-10-18T07:49:57.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:57.340+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:57.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:57.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:58.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:58.308+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:58.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:58.308+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:58.653 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:49:58.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:58.651557+0000) 2019-10-18T07:49:58.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:49:58.872+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:49:58.874148+0000) 2019-10-18T07:49:59.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:59.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:49:59.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:49:59.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:49:59.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:49:59.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:00.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:00.304+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:00.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:00.304+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:01.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:01.260+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:01.266 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:01.260+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:02.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:02.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:02.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:02.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:03.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:03.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:03.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:03.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:04.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:04.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:04.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:04.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:04.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:50:04.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:04.413461+0000) 2019-10-18T07:50:04.587 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:04.580+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:05.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:05.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:05.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:05.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:06.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:06.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:06.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:06.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:07.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:50:07.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:07.131632+0000) 2019-10-18T07:50:07.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:07.264+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:07.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:07.264+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:08.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:08.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:08.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:08.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:08.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:50:08.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:08.651763+0000) 2019-10-18T07:50:08.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:50:08.872+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:08.874270+0000) 2019-10-18T07:50:09.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:09.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:09.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:09.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:09.588 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:09.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:10.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:10.320+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:10.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:10.320+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:11.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:11.288+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:11.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:11.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:12.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:12.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:12.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:12.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:13.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:13.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:13.211 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:13.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:14.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:14.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:14.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:14.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:14.415 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:50:14.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:14.413562+0000) 2019-10-18T07:50:14.589 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:14.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:15.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:15.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:15.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:15.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:16.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:16.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:16.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:16.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:17.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:50:17.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:17.131740+0000) 2019-10-18T07:50:17.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:17.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:17.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:17.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:18.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:18.244+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:18.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:18.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:18.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:50:18.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:18.651941+0000) 2019-10-18T07:50:18.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:50:18.872+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:18.874388+0000) 2019-10-18T07:50:19.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:19.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:19.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:19.288+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:19.590 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:19.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:20.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:20.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:20.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:20.276+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:21.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:21.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:21.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:21.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:22.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:22.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:22.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:22.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:23.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:23.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:23.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:23.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:24.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:24.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:24.239 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:24.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:24.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:50:24.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:24.413664+0000) 2019-10-18T07:50:24.591 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:24.584+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:25.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:25.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:25.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:25.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:26.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:26.192+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:26.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:26.192+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:27.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:50:27.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:27.131854+0000) 2019-10-18T07:50:27.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:27.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:27.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:27.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:28.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:28.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:28.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:28.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:28.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:50:28.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:28.652118+0000) 2019-10-18T07:50:28.878 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:50:28.872+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:28.874503+0000) 2019-10-18T07:50:29.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:29.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:29.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:29.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:29.592 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:29.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:30.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:30.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:30.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:30.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:31.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:31.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:31.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:31.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:32.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:32.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:32.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:32.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:33.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:33.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:33.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:33.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:34.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:34.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:34.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:34.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:34.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:50:34.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:34.413764+0000) 2019-10-18T07:50:34.593 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:34.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:35.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:35.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:35.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:35.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:36.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:36.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:36.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:36.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:37.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:50:37.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:37.131967+0000) 2019-10-18T07:50:37.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:37.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:37.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:37.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:38.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:38.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:38.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:38.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:38.654 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:50:38.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:38.652295+0000) 2019-10-18T07:50:38.876 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:50:38.871+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:38.874622+0000) 2019-10-18T07:50:39.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:39.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:39.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:39.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:39.593 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:39.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:40.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:40.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:40.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:40.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:41.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:41.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:41.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:41.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:42.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:42.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:42.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:42.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:43.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:43.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:43.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:43.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:44.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:44.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:44.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:44.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:44.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:50:44.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:44.413863+0000) 2019-10-18T07:50:44.594 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:44.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:45.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:45.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:45.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:45.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:46.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:46.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:46.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:46.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:47.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:50:47.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:47.132087+0000) 2019-10-18T07:50:47.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:47.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:47.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:47.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:48.416 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:48.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:48.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:48.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:48.659 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:50:48.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:48.652476+0000) 2019-10-18T07:50:48.878 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:50:48.871+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:48.874744+0000) 2019-10-18T07:50:49.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:49.391+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:49.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:49.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:49.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:49.587+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:50.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:50.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:50.417 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:50.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:51.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:51.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:51.407 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:51.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:52.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:52.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:52.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:52.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:53.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:53.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:53.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:53.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:54.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:54.387+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:54.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:54.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:54.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:50:54.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:54.413962+0000) 2019-10-18T07:50:54.596 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:54.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:50:55.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:55.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:55.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:55.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:56.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:56.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:56.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:56.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:57.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:50:57.127+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:57.132214+0000) 2019-10-18T07:50:57.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:57.499+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:57.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:57.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:58.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:58.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:58.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:58.463+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:58.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:50:58.647+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:58.652655+0000) 2019-10-18T07:50:58.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:50:58.871+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:50:58.874860+0000) 2019-10-18T07:50:59.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:59.447+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:50:59.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:50:59.447+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:50:59.597 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:50:59.591+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:00.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:00.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:00.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:00.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:01.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:01.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:01.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:01.431+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:02.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:02.427+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:02.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:02.427+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:03.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:03.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:03.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:03.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:04.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:51:04.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:04.414062+0000) 2019-10-18T07:51:04.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:04.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:04.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:04.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:04.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:04.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:05.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:05.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:05.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:05.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:06.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:06.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:06.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:07.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:07.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:07.132328+0000) 2019-10-18T07:51:07.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:07.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:07.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:07.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:08.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:08.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:08.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:08.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:08.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:51:08.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:08.652835+0000) 2019-10-18T07:51:08.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:51:08.870+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:08.874978+0000) 2019-10-18T07:51:09.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:09.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:09.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:09.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:09.598 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:09.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:10.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:10.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:10.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:10.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 576 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:11.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:11.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:11.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:11.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:12.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:12.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:12.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:12.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:13.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:13.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:13.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:13.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:14.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:51:14.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:14.414162+0000) 2019-10-18T07:51:14.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:14.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:14.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:14.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:14.599 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:14.590+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:15.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:15.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:15.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:15.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:16.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:16.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:16.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:16.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:17.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:17.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:17.132441+0000) 2019-10-18T07:51:17.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:17.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:17.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:17.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:18.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:18.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:18.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:18.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:18.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:51:18.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:18.653015+0000) 2019-10-18T07:51:18.878 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:51:18.870+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:18.875095+0000) 2019-10-18T07:51:19.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:19.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:19.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:19.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:19.600 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:19.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:20.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:20.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:20.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:20.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:21.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:21.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:21.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:21.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:22.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:22.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:22.463 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:22.454+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:23.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:23.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:23.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:23.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:24.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:51:24.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:24.414263+0000) 2019-10-18T07:51:24.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:24.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:24.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:24.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:24.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:24.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:25.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:25.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:25.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:25.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:26.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:26.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:26.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:26.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:27.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:27.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:27.132550+0000) 2019-10-18T07:51:27.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:27.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:27.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:27.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:28.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:28.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:28.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:28.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:28.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:51:28.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:28.653195+0000) 2019-10-18T07:51:28.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:51:28.870+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:28.875211+0000) 2019-10-18T07:51:29.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:29.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:29.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:29.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:29.602 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:29.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:30.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:30.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:30.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:30.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:31.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:31.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:31.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:31.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:32.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:32.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:32.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:32.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:33.347 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:33.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:33.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:33.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:34.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:34.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:34.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:34.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:51:34.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:34.414362+0000) 2019-10-18T07:51:34.603 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:34.594+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:35.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:35.294+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:35.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:35.294+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:36.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:36.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:36.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:36.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:37.135 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:37.126+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:37.132653+0000) 2019-10-18T07:51:37.136 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:37.126+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:37.133532+0000) 2019-10-18T07:51:37.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:37.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:37.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:37.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:38.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:38.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:38.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:38.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:38.655 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:51:38.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:38.653408+0000) 2019-10-18T07:51:38.879 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:51:38.870+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:38.875341+0000) 2019-10-18T07:51:39.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:39.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:39.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:39.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:39.604 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:39.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:40.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:40.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:40.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:40.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:41.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:41.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:41.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:41.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:41.635 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:41.626+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:41.632874+0000) 2019-10-18T07:51:42.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:42.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:42.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:42.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:43.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:43.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:43.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:43.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:44.416 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:51:44.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:44.414462+0000) 2019-10-18T07:51:44.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:44.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:44.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:44.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:44.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:44.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:45.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:45.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:45.482 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:45.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:46.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:46.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:46.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:46.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:47.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:47.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:47.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:47.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:48.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:48.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:48.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:48.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:48.656 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:51:48.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:48.653591+0000) 2019-10-18T07:51:48.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:51:48.869+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:48.875460+0000) 2019-10-18T07:51:49.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:49.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:49.405 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:49.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:49.605 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:49.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:50.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:50.405+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:50.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:50.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:51.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:51.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:51.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:51.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:51.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:51:51.626+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:51.632986+0000) 2019-10-18T07:51:52.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:52.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:52.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:52.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:53.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:53.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:53.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:53.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:54.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:54.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:54.369 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:54.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:54.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:51:54.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:54.414564+0000) 2019-10-18T07:51:54.606 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:54.597+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:51:55.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:55.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:55.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:55.389+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:56.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:56.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:56.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:56.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:57.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:57.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:57.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:57.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:58.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:58.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:58.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:58.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:58.656 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:51:58.646+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:58.653773+0000) 2019-10-18T07:51:58.877 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:51:58.873+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:51:58.875577+0000) 2019-10-18T07:51:59.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:59.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:51:59.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:51:59.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:51:59.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:51:59.601+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:00.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:00.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:00.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:00.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:01.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:01.429+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:01.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:01.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:01.635 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:52:01.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:01.633100+0000) 2019-10-18T07:52:02.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:02.393+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:02.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:02.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:03.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:03.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:03.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:03.405+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:04.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:04.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:04.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:04.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:04.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:52:04.413+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:04.414667+0000) 2019-10-18T07:52:04.608 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:04.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:05.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:05.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:05.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:05.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:06.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:06.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:06.363 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:06.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:07.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:07.357+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:07.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:07.357+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:08.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:08.321+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:08.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:08.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:08.656 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:52:08.650+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:08.653986+0000) 2019-10-18T07:52:08.887 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:52:08.881+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:08.885297+0000) 2019-10-18T07:52:09.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:09.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:09.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:09.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:09.609 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:09.605+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:10.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:10.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:10.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:10.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:11.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:11.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:11.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:11.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:11.635 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:52:11.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:11.633216+0000) 2019-10-18T07:52:12.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:12.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:12.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:12.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:13.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:13.377+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:13.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:13.377+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:14.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:14.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:14.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:14.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:14.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:52:14.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:14.414769+0000) 2019-10-18T07:52:14.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:14.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:15.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:15.332+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:15.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:15.332+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:16.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:16.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:16.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:16.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:17.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:17.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:17.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:17.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:18.344+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:18.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:18.344+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:18.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:52:18.649+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:18.654134+0000) 2019-10-18T07:52:18.887 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:52:18.880+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:18.885423+0000) 2019-10-18T07:52:19.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:19.364+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:19.372 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:19.364+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:19.610 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:19.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:20.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:20.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:20.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:20.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:21.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:21.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:21.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:21.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:21.635 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:52:21.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:21.633330+0000) 2019-10-18T07:52:22.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:22.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:22.434 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:22.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:23.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:23.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:23.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:23.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:24.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:52:24.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:24.414871+0000) 2019-10-18T07:52:24.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:24.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:24.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:24.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:24.611 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:24.604+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:25.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:25.472+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:25.478 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:25.472+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:26.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:26.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:26.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:26.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:27.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:27.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:27.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:27.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:28.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:28.528+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:28.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:28.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:28.656 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:52:28.653+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:28.654302+0000) 2019-10-18T07:52:28.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:52:28.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:28.885674+0000) 2019-10-18T07:52:29.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:29.560+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:29.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:29.560+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:29.612 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:29.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:30.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:30.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:30.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:30.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:31.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:31.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:31.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:31.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:31.635 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:52:31.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:31.633442+0000) 2019-10-18T07:52:32.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:32.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:32.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:32.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:33.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:33.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:33.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:33.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:34.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:52:34.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:34.414972+0000) 2019-10-18T07:52:34.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:34.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:34.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:34.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:34.613 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:34.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:35.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:35.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:35.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:35.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:36.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:36.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:36.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:36.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:37.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:37.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:37.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:37.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:38.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:38.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:38.531 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:38.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:38.656 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:52:38.653+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:38.654452+0000) 2019-10-18T07:52:38.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:52:38.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:38.885792+0000) 2019-10-18T07:52:39.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:39.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:39.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:39.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:39.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:39.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:40.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:40.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:40.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:40.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:41.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:41.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:41.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:41.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:41.635 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:52:41.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:41.633556+0000) 2019-10-18T07:52:42.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:42.596+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:42.605 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:42.600+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:43.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:43.616+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:43.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:43.616+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:44.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:52:44.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:44.415067+0000) 2019-10-18T07:52:44.614 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:44.608+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:44.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:44.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:44.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:44.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:45.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:45.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:45.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:45.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:46.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:46.580+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:46.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:46.580+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:47.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:47.572+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:47.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:47.572+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:48.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:48.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:48.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:48.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:48.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:52:48.653+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:48.654641+0000) 2019-10-18T07:52:48.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:52:48.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:48.885915+0000) 2019-10-18T07:52:49.615 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:49.607+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:49.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:49.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:49.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:49.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:50.709 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:50.703+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:50.710 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:50.703+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:51.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:52:51.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:51.633669+0000) 2019-10-18T07:52:51.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:51.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:51.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:51.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:52.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:52.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:52.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:52.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:53.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:53.699+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:53.707 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:53.699+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:54.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:52:54.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:54.415169+0000) 2019-10-18T07:52:54.616 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:54.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:52:54.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:54.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:54.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:54.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:55.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:55.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:55.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:55.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:56.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:56.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:56.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:56.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:57.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:57.627+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:57.636 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:57.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:58.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:58.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:58.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:58.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:58.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:52:58.653+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:58.654805+0000) 2019-10-18T07:52:58.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:52:58.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:52:58.886029+0000) 2019-10-18T07:52:59.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:59.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:52:59.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:52:59.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:52:59.617 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:52:59.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:00.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:00.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:00.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:01.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:01.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:01.621 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:01.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:01.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:53:01.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:01.633785+0000) 2019-10-18T07:53:02.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:02.619+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:02.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:02.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:03.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:03.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:03.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:03.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:04.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:53:04.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:04.415272+0000) 2019-10-18T07:53:04.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:04.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:04.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:04.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:04.619 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:04.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:05.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:05.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:05.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:05.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:06.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:06.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:06.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:06.579+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:07.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:07.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:07.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:07.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:08.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:08.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:08.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:08.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:08.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:53:08.653+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:08.654984+0000) 2019-10-18T07:53:08.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:53:08.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:08.886149+0000) 2019-10-18T07:53:09.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:09.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:09.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:09.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:09.618 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:09.611+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:10.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:10.483+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:10.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:10.483+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:11.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:11.451+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:11.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:11.451+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:11.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:53:11.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:11.633898+0000) 2019-10-18T07:53:12.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:12.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:12.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:12.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:13.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:13.395+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:13.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:13.395+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:14.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:14.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:14.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:14.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:14.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:53:14.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:14.415373+0000) 2019-10-18T07:53:14.620 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:14.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:15.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:15.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:15.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:15.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:16.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:16.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:16.425 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:16.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:17.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:17.415+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:17.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:17.415+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:18.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:18.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:18.462 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:18.455+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:18.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:53:18.653+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:18.655171+0000) 2019-10-18T07:53:18.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:53:18.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:18.886280+0000) 2019-10-18T07:53:19.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:19.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:19.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:19.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:19.621 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:19.615+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:20.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:20.407+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:20.414 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:20.407+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:21.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:21.443+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:21.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:21.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:21.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:53:21.629+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:21.634010+0000) 2019-10-18T07:53:22.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:22.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:22.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:22.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:23.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:23.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:23.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:23.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:24.417 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:53:24.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:24.415476+0000) 2019-10-18T07:53:24.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:24.530+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:24.539 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:24.530+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:24.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:24.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:25.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:25.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:25.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:25.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:26.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:26.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:26.597 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:26.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:27.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:27.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:27.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:27.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:28.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:28.554+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:28.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:28.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:28.657 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:53:28.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:28.655350+0000) 2019-10-18T07:53:28.888 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:53:28.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:28.886397+0000) 2019-10-18T07:53:29.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:29.586+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:29.595 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:29.586+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:29.622 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:29.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:30.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:30.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:30.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:30.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:31.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:31.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:31.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:31.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:31.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:53:31.628+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:31.634109+0000) 2019-10-18T07:53:32.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:32.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:32.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:32.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:33.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:33.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:33.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:33.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:34.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:53:34.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:34.415577+0000) 2019-10-18T07:53:34.623 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:34.614+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:34.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:34.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:34.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:34.694+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:35.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:35.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:35.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:35.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:36.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:36.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:36.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:36.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:37.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:37.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:37.729 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:37.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:38.658 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:53:38.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:38.655531+0000) 2019-10-18T07:53:38.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:38.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:38.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:38.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:38.889 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:53:38.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:38.886518+0000) 2019-10-18T07:53:39.624 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:39.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:39.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:39.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:39.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:39.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:40.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:40.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:40.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:40.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:41.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:53:41.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:41.634212+0000) 2019-10-18T07:53:41.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:41.682+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:41.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:41.682+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:42.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:42.657 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:42.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:43.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:43.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:43.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:44.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:53:44.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:44.415680+0000) 2019-10-18T07:53:44.625 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:44.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:44.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:44.626+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:44.637 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:44.626+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:45.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:45.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:45.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:45.662+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:46.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:46.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:46.650 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:46.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:47.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:47.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:47.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:47.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:48.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:48.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:48.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:48.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:48.658 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:53:48.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:48.655713+0000) 2019-10-18T07:53:48.889 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:53:48.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:48.886638+0000) 2019-10-18T07:53:49.626 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:49.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:49.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:49.622+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:49.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:49.622+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:50.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:50.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:50.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:50.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:51.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:53:51.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:51.634315+0000) 2019-10-18T07:53:51.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:51.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:51.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:51.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:52.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:52.654+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:52.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:52.654+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:53.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:53.646+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:53.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:53.646+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:54.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:53:54.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:54.415783+0000) 2019-10-18T07:53:54.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:54.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:54.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:54.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:54.627 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:54.618+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:53:55.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:55.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:55.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:55.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:56.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:56.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:56.576 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:56.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:57.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:57.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:57.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:57.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:58.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:58.562+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:58.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:58.562+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:58.659 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:53:58.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:58.655894+0000) 2019-10-18T07:53:58.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:53:58.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:53:58.886755+0000) 2019-10-18T07:53:59.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:59.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:53:59.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:53:59.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:53:59.628 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:53:59.617+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:00.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:00.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:00.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:00.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:01.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:01.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:01.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:01.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:01.636 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:54:01.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:01.634427+0000) 2019-10-18T07:54:02.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:02.537+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:02.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:02.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:03.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:03.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:03.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:03.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:04.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:54:04.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:04.415885+0000) 2019-10-18T07:54:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:04.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:04.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:04.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:04.629 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:04.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:05.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:05.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:05.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:05.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:06.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:06.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:06.550 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:06.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:07.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:07.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:07.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:07.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:08.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:08.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:08.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:08.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:08.658 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:54:08.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:08.656073+0000) 2019-10-18T07:54:08.889 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:54:08.881+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:08.886873+0000) 2019-10-18T07:54:09.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:09.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:09.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:09.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:09.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:09.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:10.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:10.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:10.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:10.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:11.637 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:54:11.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:11.634531+0000) 2019-10-18T07:54:11.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:11.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:11.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:11.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:12.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:12.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:12.620 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:12.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:13.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:13.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:13.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:13.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:14.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:54:14.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:14.415985+0000) 2019-10-18T07:54:14.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:14.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:14.582 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:14.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:14.630 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:14.621+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:15.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:15.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:15.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:15.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:16.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:16.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:16.541 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:16.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:17.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:17.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:17.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:17.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:18.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:18.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:18.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:18.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:18.658 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:54:18.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:18.656254+0000) 2019-10-18T07:54:18.889 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:54:18.885+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:18.886991+0000) 2019-10-18T07:54:19.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:19.549+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:19.554 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:19.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:19.631 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:19.625+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:20.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:20.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:20.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:20.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:21.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:21.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:21.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:21.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:21.637 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:54:21.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:21.634623+0000) 2019-10-18T07:54:22.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:22.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:22.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:22.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:23.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:23.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:23.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:23.561+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:24.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:54:24.413+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:24.416088+0000) 2019-10-18T07:54:24.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:24.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:24.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:24.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:24.632 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:24.629+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:25.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:25.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:25.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:25.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:26.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:26.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:26.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:26.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:27.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:27.593+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:27.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:27.593+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:28.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:28.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:28.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:28.545+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:28.658 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:54:28.652+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:28.656462+0000) 2019-10-18T07:54:28.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:54:28.885+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:28.887110+0000) 2019-10-18T07:54:29.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:29.545+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:29.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:29.549+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:29.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:29.629+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:30.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:30.565+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:30.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:30.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:31.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:31.613+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:31.626 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:31.613+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:31.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:54:31.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:31.634735+0000) 2019-10-18T07:54:32.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:32.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:32.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:32.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:33.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:33.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:33.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:33.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:34.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:54:34.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:34.416190+0000) 2019-10-18T07:54:34.633 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:34.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:34.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:34.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:34.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:34.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:35.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:35.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:35.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:35.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:36.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:36.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:36.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:36.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:37.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:37.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:37.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:37.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:38.659 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:54:38.651+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:38.656642+0000) 2019-10-18T07:54:38.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:38.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:38.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:38.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:38.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:54:38.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:38.887228+0000) 2019-10-18T07:54:39.634 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:39.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:39.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:39.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:39.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:39.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:40.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:40.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:40.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:40.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:41.637 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:54:41.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:41.634855+0000) 2019-10-18T07:54:41.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:41.700+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:41.708 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:41.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:42.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:42.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:42.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:42.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:43.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:43.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:43.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:43.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:44.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:54:44.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:44.416294+0000) 2019-10-18T07:54:44.635 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:44.628+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:44.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:44.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:44.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:44.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:45.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:45.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:45.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:45.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:46.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:46.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:46.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:46.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:47.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:47.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:47.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:47.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:48.659 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:54:48.651+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:48.656822+0000) 2019-10-18T07:54:48.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:48.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:48.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:48.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:48.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:54:48.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:48.887349+0000) 2019-10-18T07:54:49.637 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:49.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:49.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:49.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:49.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:49.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:50.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:50.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:50.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:50.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:51.637 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:54:51.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:51.634968+0000) 2019-10-18T07:54:51.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:51.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:51.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:51.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:52.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:52.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:52.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:52.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:53.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:53.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:53.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:53.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:54.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:54:54.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:54.416398+0000) 2019-10-18T07:54:54.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:54.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:54.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:54.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:54.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:54.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:55.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:55.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:55.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:55.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:56.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:56.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:56.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:56.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:57.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:57.856+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:57.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:57.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:58.659 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:54:58.651+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:58.657032+0000) 2019-10-18T07:54:58.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:58.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:58.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:58.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:54:58.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:54:58.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:54:58.887470+0000) 2019-10-18T07:54:59.638 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:54:59.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:54:59.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:59.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:54:59.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:54:59.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:00.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:00.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:00.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:00.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:01.637 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:55:01.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:01.635079+0000) 2019-10-18T07:55:01.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:01.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:01.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:01.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:02.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:02.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:02.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:02.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:03.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:03.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:03.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:03.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:04.418 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:55:04.412+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:04.416500+0000) 2019-10-18T07:55:04.639 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:04.632+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:04.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:04.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:04.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:04.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:05.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:05.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:05.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:05.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:06.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:06.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:06.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:06.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:07.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:07.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:07.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:07.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:08.659 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:08.651+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:08.657233+0000) 2019-10-18T07:55:08.660 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:08.651+0000 7f45241c2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:08.658126+0000) 2019-10-18T07:55:08.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:08.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:08.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:08.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:08.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:55:08.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:08.887591+0000) 2019-10-18T07:55:09.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:09.631+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:09.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:09.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:09.833 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:09.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:10.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:10.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:10.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:10.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:11.637 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:55:11.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:11.635194+0000) 2019-10-18T07:55:11.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:11.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:11.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:11.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:12.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:12.823+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:12.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:12.823+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:13.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:13.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:13.157670+0000) 2019-10-18T07:55:13.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:13.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:13.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:13.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:14.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:55:14.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:14.416602+0000) 2019-10-18T07:55:14.640 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:14.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:14.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:14.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:14.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:14.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:15.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:15.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:15.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:15.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:16.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:16.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:16.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:16.811+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:17.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:17.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:17.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:17.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:18.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:18.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:18.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:18.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:18.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:55:18.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:18.887710+0000) 2019-10-18T07:55:19.641 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:19.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:19.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:19.851+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:19.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:19.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:20.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:20.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:20.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:20.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:21.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:55:21.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:21.635305+0000) 2019-10-18T07:55:21.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:21.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:21.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:21.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:22.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:22.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:22.902 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:22.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:23.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:23.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:23.157881+0000) 2019-10-18T07:55:23.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:23.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:23.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:23.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:24.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:55:24.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:24.416705+0000) 2019-10-18T07:55:24.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:24.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:24.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:24.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:24.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:24.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:25.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:25.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:25.929 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:25.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:26.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:26.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:26.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:26.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:27.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:27.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:27.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:27.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:28.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:28.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:28.874 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:28.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:28.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:55:28.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:28.887828+0000) 2019-10-18T07:55:29.642 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:29.635+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:29.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:29.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:29.854 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:29.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:30.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:30.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:30.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:30.859+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:31.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:55:31.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:31.635403+0000) 2019-10-18T07:55:31.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:31.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:31.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:31.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:32.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:32.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:32.927 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:32.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:33.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:33.151+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:33.158092+0000) 2019-10-18T07:55:33.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:33.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:33.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:33.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:34.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:55:34.411+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:34.416806+0000) 2019-10-18T07:55:34.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:34.639+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:34.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:34.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:34.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:35.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:35.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:35.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:35.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:36.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:36.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:36.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:36.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:37.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:37.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:37.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:37.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:38.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:55:38.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:38.887953+0000) 2019-10-18T07:55:38.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:38.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:38.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:38.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:39.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:39.639+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:39.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:39.971+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:39.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:39.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:40.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:40.943+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:40.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:40.943+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:41.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:55:41.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:41.635516+0000) 2019-10-18T07:55:41.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:41.983+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:41.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:41.983+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:43.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:43.015+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:43.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:43.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:43.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:43.155+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:43.158261+0000) 2019-10-18T07:55:44.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:44.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:44.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:44.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:44.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:55:44.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:44.416905+0000) 2019-10-18T07:55:44.645 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:44.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:45.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:45.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:45.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:45.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:46.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:46.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:46.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:46.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:47.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:47.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:47.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:47.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:48.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:48.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:48.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:48.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:48.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:55:48.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:48.888074+0000) 2019-10-18T07:55:48.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:48.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:48.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:48.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:49.646 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:49.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:50.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:50.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:50.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:51.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:51.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:51.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:51.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:51.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:55:51.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:51.635629+0000) 2019-10-18T07:55:52.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:52.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:52.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:52.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:53.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:53.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:53.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:53.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:53.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:55:53.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:53.158408+0000) 2019-10-18T07:55:54.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:54.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:54.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:54.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:54.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:55:54.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:54.417010+0000) 2019-10-18T07:55:54.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:54.638+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:55:55.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:55.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:55.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:55.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:56.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:56.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:56.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:56.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:57.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:57.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:57.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:57.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:58.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:58.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:58.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:58.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:58.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:55:58.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:55:58.888189+0000) 2019-10-18T07:55:59.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:59.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:55:59.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:55:59.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:55:59.648 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:55:59.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:00.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:00.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:00.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:00.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:01.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:01.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:01.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:01.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:01.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:56:01.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:01.635730+0000) 2019-10-18T07:56:02.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:02.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:02.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:02.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:03.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:03.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:03.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:03.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:03.160 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:56:03.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:03.158583+0000) 2019-10-18T07:56:04.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:04.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:04.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:04.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:04.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:56:04.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:04.417118+0000) 2019-10-18T07:56:04.650 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:04.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:05.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:05.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:05.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:05.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:06.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:06.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:06.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:06.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:07.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:07.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:07.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:07.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:08.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:08.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:08.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:08.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:08.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:56:08.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:08.888304+0000) 2019-10-18T07:56:09.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:09.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:09.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:09.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:09.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:09.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:10.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:10.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:10.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:10.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:11.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:11.146+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:11.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:11.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:11.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:56:11.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:11.635844+0000) 2019-10-18T07:56:12.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:12.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:12.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:12.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:13.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:56:13.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:13.158757+0000) 2019-10-18T07:56:13.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:13.214+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:13.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:13.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:14.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:14.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:14.189 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:14.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:14.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:56:14.410+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:14.417220+0000) 2019-10-18T07:56:14.651 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:14.642+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:15.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:15.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:15.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:15.206+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:16.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:16.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:16.172 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:16.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:17.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:17.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:17.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:17.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:18.138 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:18.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:18.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:18.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:18.890 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:56:18.882+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:18.888421+0000) 2019-10-18T07:56:19.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:19.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:19.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:19.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:19.652 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:19.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:20.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:20.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:20.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:20.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:21.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:21.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:21.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:21.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:21.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:56:21.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:21.635954+0000) 2019-10-18T07:56:22.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:22.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:22.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:22.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:23.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:23.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:23.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:23.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:23.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:56:23.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:23.158965+0000) 2019-10-18T07:56:24.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:24.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:24.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:24.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:24.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:56:24.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:24.417320+0000) 2019-10-18T07:56:24.653 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:24.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:25.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:25.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:25.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:25.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:26.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:26.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:26.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:26.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:27.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:27.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:27.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:27.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:28.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:28.045+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:28.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:28.049+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:28.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:56:28.881+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:28.888538+0000) 2019-10-18T07:56:29.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:29.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:29.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:29.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:29.654 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:29.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:30.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:30.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:30.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:30.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:31.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:31.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:31.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:31.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:31.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:56:31.630+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:31.636065+0000) 2019-10-18T07:56:32.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:32.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:32.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:32.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:33.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:33.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:33.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:33.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:56:33.154+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:33.159173+0000) 2019-10-18T07:56:34.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:34.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:34.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:34.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:34.419 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:56:34.409+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:34.417417+0000) 2019-10-18T07:56:34.655 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:34.645+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:35.104 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:35.093+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:35.105 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:35.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:36.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:36.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:36.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:36.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:37.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:37.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:37.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:37.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:38.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:38.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:38.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:38.129+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:38.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:56:38.885+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:38.888655+0000) 2019-10-18T07:56:39.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:39.113+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:39.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:39.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:39.656 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:39.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:40.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:40.109+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:40.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:40.109+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:41.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:41.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:41.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:41.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:41.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:56:41.634+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:41.636179+0000) 2019-10-18T07:56:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:42.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:42.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:42.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:43.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:43.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:43.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:43.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:43.161 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:56:43.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:43.159353+0000) 2019-10-18T07:56:44.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:44.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:44.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:44.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:44.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:56:44.417+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:44.417524+0000) 2019-10-18T07:56:44.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:44.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:45.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:45.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:45.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:45.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:46.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:46.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:46.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:46.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:47.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:47.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:47.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:47.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:48.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:48.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:48.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:48.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:48.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:56:48.885+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:48.888768+0000) 2019-10-18T07:56:49.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:49.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:49.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:49.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:49.657 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:49.653+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:50.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:50.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:50.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:50.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:51.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:51.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:51.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:51.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:51.638 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:56:51.634+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:51.636292+0000) 2019-10-18T07:56:52.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:52.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:52.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:52.201+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:53.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:56:53.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:53.159566+0000) 2019-10-18T07:56:53.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:53.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:53.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:53.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:54.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:54.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:54.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:54.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:54.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:56:54.416+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:54.417630+0000) 2019-10-18T07:56:54.658 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:54.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:56:55.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:55.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:55.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:55.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:56.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:56.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:56.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:56.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:57.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:57.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:57.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:57.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:58.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:58.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:58.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:58.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:58.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:56:58.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:56:58.888885+0000) 2019-10-18T07:56:59.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:59.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:56:59.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:56:59.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:56:59.659 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:56:59.652+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:00.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:00.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:00.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:00.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:01.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:01.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:01.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:01.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:01.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:57:01.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:01.636403+0000) 2019-10-18T07:57:02.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:02.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:02.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:02.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:03.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:03.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:03.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:03.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:03.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:57:03.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:03.159747+0000) 2019-10-18T07:57:03.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:03.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:03.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:03.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:04.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:57:04.416+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:04.417731+0000) 2019-10-18T07:57:04.661 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:04.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:04.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:04.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:04.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:04.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:05.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:05.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:05.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:05.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:06.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:06.876+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:06.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:06.876+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:07.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:07.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:07.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:07.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:08.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:08.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:08.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:08.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:08.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:57:08.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:08.889000+0000) 2019-10-18T07:57:09.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:09.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:09.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:09.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:09.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:09.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:10.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:10.852+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:10.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:10.856+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:11.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:57:11.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:11.636518+0000) 2019-10-18T07:57:11.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:11.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:46:40.092450+0000 (oldest deadline 2019-10-18T07:47:00.092450+0000) 2019-10-18T07:57:11.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:11.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:12.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:12.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:13.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:57:13.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:13.159925+0000) 2019-10-18T07:57:13.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:13.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:14.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:57:14.416+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:14.417834+0000) 2019-10-18T07:57:14.662 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:14.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:14.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:14.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:15.881 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:15.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:16.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:16.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:17.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:17.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:18.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:18.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:18.897 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:57:18.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:18.889121+0000) 2019-10-18T07:57:19.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:19.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:19.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:19.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:20.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:20.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:21.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:57:21.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:21.636630+0000) 2019-10-18T07:57:21.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:21.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:22.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:22.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:23.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:57:23.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:23.160103+0000) 2019-10-18T07:57:23.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:23.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:24.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:57:24.416+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:24.417930+0000) 2019-10-18T07:57:24.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:24.656+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:24.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:24.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:25.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:25.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:26.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:26.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:27.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:27.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:28.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:28.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:28.891 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:57:28.884+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:28.889238+0000) 2019-10-18T07:57:29.664 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:29.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:29.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:29.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:30.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:30.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:31.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:57:31.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:31.636750+0000) 2019-10-18T07:57:31.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:31.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:32.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:32.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:33.162 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:57:33.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:33.160288+0000) 2019-10-18T07:57:33.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:33.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:34.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:57:34.415+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:34.418030+0000) 2019-10-18T07:57:34.665 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:34.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:34.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:34.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:34.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:34.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:35.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:35.759+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:35.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:35.759+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:36.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:36.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:36.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:36.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:37.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:37.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:37.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:37.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:38.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:38.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:38.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:38.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:38.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:57:38.883+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:38.889353+0000) 2019-10-18T07:57:39.666 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:39.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:39.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:39.779+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:39.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:39.779+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:40.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:40.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:40.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:40.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:41.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:57:41.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:41.636852+0000) 2019-10-18T07:57:41.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:41.819+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:41.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:41.819+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:42.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:42.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:42.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:42.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:43.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:57:43.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:43.160508+0000) 2019-10-18T07:57:43.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:43.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:43.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:43.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:44.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:57:44.415+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:44.418135+0000) 2019-10-18T07:57:44.667 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:44.659+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:44.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:44.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:44.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:44.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:45.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:45.859+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:45.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:45.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:46.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:46.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:46.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:46.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:47.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:47.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:47.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:47.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:48.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:57:48.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:48.889500+0000) 2019-10-18T07:57:48.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:48.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:48.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:48.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:49.668 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:49.663+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:49.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:49.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:49.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:49.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:50.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:50.907+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:50.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:50.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:51.644 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:57:51.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:51.636976+0000) 2019-10-18T07:57:51.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:51.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:51.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:51.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:52.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:52.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:52.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:52.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:53.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:57:53.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:53.160692+0000) 2019-10-18T07:57:53.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:53.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:53.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:53.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:54.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:57:54.415+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:54.418242+0000) 2019-10-18T07:57:54.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:54.663+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:54.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:54.911+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:54.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:54.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:55.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:55.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:55.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:55.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:56.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:56.967+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:56.977 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:56.971+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:57.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:57.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:57.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:57.927+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:58.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:57:58.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:57:58.889628+0000) 2019-10-18T07:57:58.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:58.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:58.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:58.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:57:59.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:57:59.663+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:57:59.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:59.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:57:59.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:57:59.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:00.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:00.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:00.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:00.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:01.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:58:01.633+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:01.637092+0000) 2019-10-18T07:58:01.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:01.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:01.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:01.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:02.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:02.919+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:02.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:02.919+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:03.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:58:03.157+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:03.160879+0000) 2019-10-18T07:58:03.937 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:03.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:03.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:03.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:04.420 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:58:04.414+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:04.418344+0000) 2019-10-18T07:58:04.670 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:04.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:04.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:04.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:04.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:04.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:05.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:05.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:05.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:05.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:06.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:06.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:06.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:06.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:07.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:07.903 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:07.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:08.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:58:08.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:08.889739+0000) 2019-10-18T07:58:08.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:08.930+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:08.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:08.934+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:09.671 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:09.662+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:09.890+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:09.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:09.890+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:10.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:10.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:10.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:10.866+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:11.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:58:11.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:11.637206+0000) 2019-10-18T07:58:11.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:11.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:11.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:11.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:12.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:12.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:12.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:12.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:13.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:58:13.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:13.161058+0000) 2019-10-18T07:58:13.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:13.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:13.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:13.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:14.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:58:14.414+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:14.418451+0000) 2019-10-18T07:58:14.672 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:14.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:14.834+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:14.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:14.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:15.845 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:15.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:15.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:15.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:16.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:16.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:16.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:16.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:17.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:17.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:17.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:17.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:18.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:18.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:18.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:18.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:18.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:58:18.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:18.889856+0000) 2019-10-18T07:58:19.673 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:19.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:19.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:19.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:19.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:19.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:20.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:20.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:20.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:20.766+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:21.639 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:58:21.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:21.637321+0000) 2019-10-18T07:58:21.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:21.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:21.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:21.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:22.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:22.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:22.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:22.782+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:23.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:58:23.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:23.161238+0000) 2019-10-18T07:58:23.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:23.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:23.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:23.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:24.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:58:24.414+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:24.418557+0000) 2019-10-18T07:58:24.674 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:24.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:24.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:24.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:24.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:24.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:25.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:25.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:25.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:25.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:26.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:26.742+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:26.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:26.742+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:27.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:27.754+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:27.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:27.754+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:28.799 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:28.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:28.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:28.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:28.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:58:28.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:28.889972+0000) 2019-10-18T07:58:29.675 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:29.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:29.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:29.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:29.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:29.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:30.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:30.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:30.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:30.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:31.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:58:31.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:31.637439+0000) 2019-10-18T07:58:31.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:31.786+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:31.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:31.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:32.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:32.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:32.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:32.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:33.163 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:58:33.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:33.161418+0000) 2019-10-18T07:58:33.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:33.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:33.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:33.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:34.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:58:34.414+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:34.418659+0000) 2019-10-18T07:58:34.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:34.666+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:34.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:34.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:34.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:34.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:35.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:35.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:35.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:35.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:36.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:36.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:36.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:36.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:37.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:37.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:37.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:37.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:38.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:38.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:38.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:38.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:38.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:58:38.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:38.890095+0000) 2019-10-18T07:58:39.676 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:39.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:39.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:39.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:39.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:39.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:40.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:40.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:40.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:40.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:41.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:58:41.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:41.637544+0000) 2019-10-18T07:58:41.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:41.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:41.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:41.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:42.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:42.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:42.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:42.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:43.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:58:43.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:43.161601+0000) 2019-10-18T07:58:43.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:43.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:43.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:43.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:44.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:58:44.413+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:44.418764+0000) 2019-10-18T07:58:44.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:44.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:44.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:44.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:44.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:44.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:45.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:45.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:45.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:45.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:46.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:46.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:46.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:46.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:47.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:47.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:47.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:47.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:48.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:48.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:48.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:48.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:48.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:58:48.885+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:48.890218+0000) 2019-10-18T07:58:49.678 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:49.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:49.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:49.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:49.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:49.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:50.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:50.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:50.829 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:50.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:51.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:58:51.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:51.637646+0000) 2019-10-18T07:58:51.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:51.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:51.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:51.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:52.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:52.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:52.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:52.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:53.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:58:53.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:53.161814+0000) 2019-10-18T07:58:53.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:53.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:53.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:53.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:54.421 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:58:54.413+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:54.418864+0000) 2019-10-18T07:58:54.679 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:54.669+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:54.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:54.853+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:54.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:54.853+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:55.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:55.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:55.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:55.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:56.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:56.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:56.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:56.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:57.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:57.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:57.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:57.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:58.892 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:58:58.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:58:58.890332+0000) 2019-10-18T07:58:58.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:58.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:58.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:58.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:58:59.680 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:58:59.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:58:59.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:59.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:58:59.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:58:59.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:00.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:00.901+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:00.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:00.901+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:01.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:59:01.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:01.637755+0000) 2019-10-18T07:59:01.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:01.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:01.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:01.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:02.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:02.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:02.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:02.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:03.164 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:59:03.156+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:03.161992+0000) 2019-10-18T07:59:03.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:03.813+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:03.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:03.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:04.423 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:04.417+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:04.418960+0000) 2019-10-18T07:59:04.423 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:04.417+0000 7ff3a7540700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:04.420029+0000) 2019-10-18T07:59:04.681 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:04.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:04.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:04.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:04.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:04.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:05.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:05.789+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:05.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:05.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:06.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:06.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:06.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:06.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:07.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:07.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:07.863 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:07.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:08.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:08.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:08.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:08.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:08.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:59:08.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:08.890449+0000) 2019-10-18T07:59:08.921 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:08.917+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:08.919216+0000) 2019-10-18T07:59:09.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:09.677+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:09.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:09.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:09.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:09.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:10.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:10.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:10.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:10.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:11.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:59:11.632+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:11.637868+0000) 2019-10-18T07:59:11.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:11.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:11.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:11.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:12.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:12.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:12.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:12.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:13.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:59:13.160+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:13.162198+0000) 2019-10-18T07:59:13.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:13.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:13.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:13.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:14.682 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:14.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:15.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:15.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:15.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:15.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:16.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:16.028+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:16.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:16.028+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:17.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:17.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:17.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:17.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:18.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:18.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:18.027 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:18.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:18.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:59:18.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:18.890564+0000) 2019-10-18T07:59:18.921 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:18.916+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:18.919312+0000) 2019-10-18T07:59:19.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:19.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:19.053 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:19.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:19.683 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:19.676+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:20.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:20.044+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:20.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:20.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:21.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:21.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:21.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:21.096+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:21.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:59:21.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:21.637983+0000) 2019-10-18T07:59:22.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:22.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:22.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:22.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:23.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:23.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:23.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:23.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:23.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:59:23.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:23.162370+0000) 2019-10-18T07:59:24.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:24.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:24.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:24.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:24.684 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:24.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:25.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:25.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:25.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:25.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:25.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:25.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:25.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:25.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:26.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:26.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:26.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:26.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:27.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:27.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:27.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:27.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:28.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:59:28.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:28.890676+0000) 2019-10-18T07:59:28.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:28.916+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:28.919406+0000) 2019-10-18T07:59:28.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:28.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:28.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:28.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:29.685 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:29.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:29.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:29.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:29.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:29.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:30.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:30.980+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:30.988 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:30.980+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:31.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:59:31.631+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:31.638102+0000) 2019-10-18T07:59:31.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:31.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:31.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:31.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:32.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:32.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:32.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:33.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:59:33.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:33.162550+0000) 2019-10-18T07:59:33.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:33.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:33.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:33.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:34.686 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:34.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:34.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:34.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:34.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:34.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:35.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:35.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:35.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:35.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:37.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:37.000+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:37.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:37.000+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:37.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:37.968+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:37.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:37.968+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:38.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:59:38.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:38.890790+0000) 2019-10-18T07:59:38.922 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:38.916+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:38.919500+0000) 2019-10-18T07:59:39.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:39.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:39.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:39.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:39.687 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:39.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:40.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:40.008+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:40.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:40.008+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:41.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:41.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:41.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:41.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:41.645 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:59:41.635+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:41.638226+0000) 2019-10-18T07:59:42.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:42.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:42.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:42.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:43.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:43.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:43.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:43.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:43.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:59:43.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:43.162745+0000) 2019-10-18T07:59:44.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:44.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:44.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:44.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:44.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:44.680+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:45.119 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:45.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:45.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:45.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:46.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:46.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:46.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:46.148+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:47.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:47.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:47.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:47.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:48.220+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:48.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:48.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:48.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:59:48.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:48.890911+0000) 2019-10-18T07:59:48.922 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:48.916+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:48.919598+0000) 2019-10-18T07:59:49.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:49.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:49.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:49.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:49.688 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:49.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:50.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:50.163+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:50.171 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:50.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:51.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:51.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:51.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:51.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:51.640 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T07:59:51.635+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:51.638334+0000) 2019-10-18T07:59:52.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:52.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:52.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:52.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:53.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T07:59:53.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:53.162931+0000) 2019-10-18T07:59:53.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:53.287+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:53.297 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:53.287+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:54.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:54.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:54.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:54.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:54.689 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:54.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T07:59:55.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:55.283+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:55.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:55.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:56.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:56.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:56.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:56.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:57.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:57.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:57.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:57.327+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:58.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:58.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:58.361 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:58.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:58.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T07:59:58.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:58.891028+0000) 2019-10-18T07:59:58.922 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T07:59:58.915+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T06:59:58.919695+0000) 2019-10-18T07:59:59.364 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:59.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T07:59:59.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T07:59:59.355+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T07:59:59.690 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T07:59:59.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:00.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:00.351 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:00.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:01.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:01.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:01.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:01.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:01.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:01.635+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:01.638440+0000) 2019-10-18T08:00:02.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:02.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:02.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:02.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:03.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:00:03.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:03.163135+0000) 2019-10-18T08:00:03.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:03.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:03.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:03.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:04.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:04.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:04.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:04.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:04.691 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:04.683+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:05.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:05.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:05.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:05.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:06.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:06.175+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:06.183 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:06.175+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:07.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:07.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:07.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:07.195+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:08.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:08.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:08.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:08.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:08.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:00:08.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:08.891144+0000) 2019-10-18T08:00:08.922 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:00:08.915+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:08.919797+0000) 2019-10-18T08:00:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:09.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:09.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:09.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:09.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:09.687+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:10.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:10.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:10.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:11.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:11.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:11.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:11.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:11.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:11.635+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:11.638550+0000) 2019-10-18T08:00:12.268 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:12.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:12.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:12.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:13.165 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:00:13.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:13.163345+0000) 2019-10-18T08:00:13.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:13.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:13.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:13.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:14.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:14.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:14.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:14.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:14.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:14.687+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:15.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:15.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:15.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:15.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:16.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:16.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:16.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:16.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:17.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:17.171+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:17.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:17.171+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:18.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:18.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:18.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:18.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:18.893 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:00:18.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:18.891279+0000) 2019-10-18T08:00:18.923 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:00:18.915+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:18.919893+0000) 2019-10-18T08:00:19.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:19.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:19.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:19.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:19.693 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:19.687+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:20.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:20.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:20.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:20.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:21.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:21.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:21.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:21.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:21.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:21.635+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:21.638650+0000) 2019-10-18T08:00:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:22.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:22.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:22.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:23.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:00:23.159+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:23.163524+0000) 2019-10-18T08:00:23.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:23.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:23.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:23.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:24.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:24.278+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:24.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:24.278+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:24.694 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:24.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:25.294 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:25.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:25.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:25.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:26.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:26.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:26.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:26.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:27.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:27.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:27.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:27.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:28.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:28.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:28.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:28.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:28.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:00:28.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:28.891397+0000) 2019-10-18T08:00:28.922 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:00:28.914+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:28.919988+0000) 2019-10-18T08:00:29.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:29.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:29.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:29.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:29.695 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:29.686+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:30.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:30.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:30.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:30.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:31.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:31.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:31.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:31.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:31.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:31.634+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:31.638745+0000) 2019-10-18T08:00:31.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:31.634+0000 7f552170b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:31.639652+0000) 2019-10-18T08:00:32.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:32.370+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:32.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:32.370+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:33.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:00:33.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:33.163702+0000) 2019-10-18T08:00:33.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:33.374+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:33.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:33.374+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:34.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:34.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:34.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:34.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:34.696 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:34.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:35.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:35.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:35.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:35.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:36.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:36.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:36.138956+0000) 2019-10-18T08:00:36.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:36.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:36.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:36.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:37.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:37.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:37.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:37.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:38.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:38.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:38.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:38.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:38.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:00:38.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:38.891512+0000) 2019-10-18T08:00:38.922 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:00:38.914+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:38.920085+0000) 2019-10-18T08:00:39.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:39.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:39.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:39.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:39.697 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:39.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:40.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:40.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:40.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:40.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:41.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:41.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:41.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:41.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:42.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:42.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:42.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:42.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:43.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:00:43.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:43.163882+0000) 2019-10-18T08:00:43.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:43.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:43.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:43.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:44.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:44.262+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:44.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:44.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:44.705 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:44.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:45.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:45.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:45.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:45.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:46.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:46.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:46.139065+0000) 2019-10-18T08:00:46.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:46.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:46.273 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:46.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:47.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:47.242+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:47.250 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:47.242+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:48.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:48.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:48.293 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:48.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:48.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:00:48.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:48.891632+0000) 2019-10-18T08:00:48.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:00:48.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:48.921516+0000) 2019-10-18T08:00:49.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:49.318+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:49.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:49.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:49.699 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:49.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:50.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:50.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:50.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:50.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:51.314 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:51.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:51.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:51.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:52.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:52.330+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:52.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:52.330+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:53.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:00:53.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:53.164077+0000) 2019-10-18T08:00:53.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:53.310+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:53.320 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:53.310+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:54.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:54.338+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:54.350 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:54.338+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:54.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:54.690+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:00:55.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:55.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:55.322 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:55.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:56.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:00:56.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:56.139177+0000) 2019-10-18T08:00:56.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:56.362+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:56.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:56.362+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:57.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:57.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:57.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:57.402+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:58.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:58.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:58.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:58.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:58.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:00:58.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:58.891750+0000) 2019-10-18T08:00:58.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:00:58.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:00:58.921613+0000) 2019-10-18T08:00:59.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:59.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:00:59.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:00:59.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:00:59.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:00:59.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:00.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:00.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:00.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:00.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:01.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:01.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:01.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:02.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:02.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:02.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:02.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:03.172 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:01:03.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:03.164262+0000) 2019-10-18T08:01:03.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:03.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:03.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:03.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:04.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:04.373+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:04.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:04.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:04.701 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:04.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:05.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:05.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:05.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:05.329+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:06.141 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:01:06.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:06.139288+0000) 2019-10-18T08:01:06.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:06.365+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:06.375 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:06.365+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:07.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:07.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:07.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:07.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:08.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:08.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:08.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:08.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:08.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:01:08.885+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:08.891865+0000) 2019-10-18T08:01:08.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:01:08.917+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:08.921713+0000) 2019-10-18T08:01:09.483 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:09.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:09.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:09.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:09.702 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:09.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:10.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:10.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:10.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:10.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:11.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:11.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:11.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:11.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:12.480 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:12.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:12.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:12.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:13.166 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:01:13.158+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:13.164442+0000) 2019-10-18T08:01:13.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:13.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:13.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:13.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:14.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:14.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:14.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:14.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:14.709 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:14.693+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:15.514 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:15.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:15.515 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:15.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:16.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:01:16.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:16.139402+0000) 2019-10-18T08:01:16.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:16.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:16.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:16.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:17.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:17.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:17.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:17.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:18.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:18.441+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:18.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:18.441+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:18.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:01:18.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:18.891984+0000) 2019-10-18T08:01:18.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:01:18.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:18.921809+0000) 2019-10-18T08:01:19.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:19.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:19.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:19.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:19.704 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:19.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:20.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:20.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:20.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:20.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:21.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:21.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:21.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:21.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:22.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:22.473+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:22.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:22.473+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:23.169 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:01:23.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:23.164629+0000) 2019-10-18T08:01:23.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:23.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:23.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:23.461+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:24.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:24.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:24.481 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:24.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:24.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:24.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:25.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:25.449+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:25.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:25.449+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:26.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:01:26.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:26.139518+0000) 2019-10-18T08:01:26.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:26.445+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:26.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:26.445+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:27.458 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:27.453+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:27.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:27.453+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:28.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:28.469+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:28.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:28.469+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:28.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:01:28.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:28.892100+0000) 2019-10-18T08:01:28.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:01:28.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:28.921917+0000) 2019-10-18T08:01:29.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:29.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:29.522 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:29.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:29.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:29.701+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:30.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:30.541+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:30.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:30.541+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:31.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:31.493+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:31.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:31.493+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:32.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:32.505+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:32.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:32.505+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:33.167 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:01:33.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:33.164838+0000) 2019-10-18T08:01:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:33.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:33.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:33.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:34.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:34.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:34.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:34.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:34.706 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:34.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:35.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:35.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:35.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:35.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:36.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:01:36.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:36.139632+0000) 2019-10-18T08:01:36.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:36.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:36.572 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:36.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:37.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:37.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:37.547 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:37.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:38.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:38.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:38.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:38.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:38.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:01:38.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:38.892211+0000) 2019-10-18T08:01:38.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:01:38.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:38.922015+0000) 2019-10-18T08:01:39.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:39.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:39.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:39.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:39.707 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:39.700+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:40.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:40.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:40.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:40.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:41.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:41.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:41.495 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:41.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:42.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:42.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:42.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:42.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:43.167 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:01:43.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:43.165020+0000) 2019-10-18T08:01:43.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:43.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:43.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:43.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:44.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:44.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:44.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:44.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:44.708 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:44.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:45.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:45.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:45.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:45.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:46.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:01:46.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:46.139744+0000) 2019-10-18T08:01:46.505 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:46.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:46.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:46.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:47.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:47.552+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:47.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:47.552+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:48.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:48.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:48.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:48.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:48.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:01:48.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:48.892329+0000) 2019-10-18T08:01:48.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:01:48.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:48.922111+0000) 2019-10-18T08:01:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:49.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:49.540 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:49.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:49.710 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:49.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:50.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:50.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:50.517 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:50.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:51.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:51.476+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:51.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:51.476+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:52.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:52.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:52.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:52.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:53.168 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:01:53.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:53.165201+0000) 2019-10-18T08:01:53.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:53.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:53.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:53.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:54.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:54.356+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:54.362 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:54.356+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:54.711 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:54.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:01:55.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:55.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:55.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:55.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:56.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:01:56.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:56.139858+0000) 2019-10-18T08:01:56.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:56.428+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:56.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:56.428+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:57.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:57.384+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:57.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:57.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:58.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:58.412+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:58.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:58.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:58.894 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:01:58.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:58.892444+0000) 2019-10-18T08:01:58.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:01:58.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:01:58.922208+0000) 2019-10-18T08:01:59.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:59.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:01:59.472 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:01:59.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:01:59.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:01:59.704+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:00.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:00.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:00.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:00.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:01.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:01.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:01.543 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:01.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:02.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:02.520+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:02.526 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:02.520+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:03.168 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:02:03.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:03.165386+0000) 2019-10-18T08:02:03.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:03.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:03.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:03.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:04.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:04.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:04.512 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:04.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:04.712 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:04.708+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:05.532 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:05.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:05.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:05.528+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:06.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:02:06.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:06.139971+0000) 2019-10-18T08:02:06.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:06.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:06.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:06.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:07.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:07.508+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:07.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:07.508+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:08.499 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:08.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:08.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:08.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:08.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:02:08.888+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:08.892563+0000) 2019-10-18T08:02:08.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:02:08.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:08.922304+0000) 2019-10-18T08:02:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:09.503+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:09.511 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:09.503+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:09.713 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:09.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:10.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:10.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:10.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:10.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:11.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:11.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:11.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:11.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:12.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:12.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:12.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:12.623+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:13.168 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:02:13.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:13.165596+0000) 2019-10-18T08:02:13.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:13.575+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:13.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:13.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:14.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:14.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:14.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:14.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:14.714 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:14.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:15.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:15.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:15.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:15.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:16.142 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:02:16.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:16.140086+0000) 2019-10-18T08:02:16.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:16.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:16.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:16.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:17.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:17.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:17.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:17.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:18.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:18.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:18.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:18.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:18.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:02:18.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:18.892693+0000) 2019-10-18T08:02:18.924 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:02:18.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:18.922398+0000) 2019-10-18T08:02:19.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:19.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:19.613 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:19.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:19.715 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:19.707+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:20.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:20.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:20.663 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:20.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:21.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:21.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:21.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:21.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:22.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:22.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:22.665 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:22.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:23.168 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:02:23.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:23.165781+0000) 2019-10-18T08:02:23.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:23.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:23.651 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:23.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:24.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:24.643+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:24.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:24.643+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:24.717 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:24.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:25.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:25.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:25.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:25.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:26.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:02:26.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:26.140202+0000) 2019-10-18T08:02:26.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:26.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:26.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:26.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:27.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:27.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:27.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:27.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:28.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:28.651+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:28.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:28.651+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:28.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:02:28.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:28.892812+0000) 2019-10-18T08:02:28.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:02:28.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:28.922496+0000) 2019-10-18T08:02:29.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:29.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:29.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:29.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:29.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:29.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:30.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:30.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:30.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:30.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:31.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:31.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:31.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:31.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:32.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:32.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:32.717 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:32.711+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:33.168 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:02:33.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:33.165958+0000) 2019-10-18T08:02:33.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:33.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:33.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:33.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:34.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:34.655+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:34.664 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:34.655+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:34.718 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:34.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:35.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:35.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:35.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:35.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:36.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:02:36.137+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:36.140321+0000) 2019-10-18T08:02:36.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:36.727+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:36.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:36.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:37.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:37.735+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:37.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:37.739+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:38.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:38.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:38.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:38.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:38.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:02:38.887+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:38.892945+0000) 2019-10-18T08:02:38.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:02:38.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:38.922597+0000) 2019-10-18T08:02:39.720 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:39.711+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:39.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:39.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:39.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:39.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:40.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:40.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:40.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:40.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:41.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:41.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:41.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:41.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:42.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:42.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:42.751 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:42.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:43.169 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:02:43.161+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:43.166142+0000) 2019-10-18T08:02:43.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:43.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:43.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:43.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:44.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:44.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:44.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:44.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:44.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:44.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:45.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:45.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:45.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:45.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:46.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:02:46.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:46.140443+0000) 2019-10-18T08:02:46.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:46.770+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:46.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:46.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:47.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:47.774+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:47.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:47.774+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:48.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:48.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:48.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:48.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:48.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:02:48.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:48.893068+0000) 2019-10-18T08:02:48.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:02:48.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:48.922695+0000) 2019-10-18T08:02:49.721 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:49.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:49.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:49.826+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:49.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:49.830+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:50.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:50.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:50.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:50.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:51.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:51.898+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:51.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:51.898+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:52.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:52.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:52.861 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:52.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:53.168 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:02:53.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:53.166327+0000) 2019-10-18T08:02:53.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:53.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:53.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:53.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:54.722 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:54.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:54.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:54.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:54.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:54.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:55.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:55.838+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:55.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:55.838+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:56.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:02:56.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:56.140559+0000) 2019-10-18T08:02:56.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:56.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:56.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:56.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:57.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:57.794+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:57.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:57.794+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:58.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:58.758+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:58.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:58.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:02:58.895 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:02:58.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:58.893183+0000) 2019-10-18T08:02:58.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:02:58.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:02:58.922791+0000) 2019-10-18T08:02:59.723 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:02:59.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:02:59.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:59.810+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:02:59.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:02:59.810+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:00.867 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:00.858+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:00.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:00.858+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:01.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:01.862+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:01.871 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:01.862+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:02.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:02.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:02.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:02.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:03.169 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:03:03.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:03.166476+0000) 2019-10-18T08:03:03.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:03.846+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:03.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:03.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:04.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:04.714+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:04.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:04.802+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:04.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:04.802+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:05.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:05.762+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:05.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:05.762+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:06.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:03:06.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:06.140674+0000) 2019-10-18T08:03:06.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:06.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:06.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:06.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:07.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:07.814+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:07.821 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:07.814+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:08.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:08.850+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:08.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:08.850+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:08.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:03:08.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:08.893301+0000) 2019-10-18T08:03:08.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:03:08.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:08.922886+0000) 2019-10-18T08:03:09.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:09.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:09.878 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:09.870+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:09.879 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:09.870+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:10.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:10.878+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:10.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:10.878+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:11.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:11.854+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:11.862 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:11.854+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:12.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:12.830+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:12.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:12.834+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:13.169 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:03:13.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:13.166643+0000) 2019-10-18T08:03:13.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:13.806+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:13.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:13.806+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:14.725 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:14.718+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:14.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:14.778+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:14.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:14.778+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:15.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:15.766+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:15.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:15.770+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:16.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:03:16.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:16.140787+0000) 2019-10-18T08:03:16.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:16.790+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:16.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:16.790+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:17.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:17.818+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:17.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:17.818+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:18.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:18.842+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:18.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:18.842+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:18.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:03:18.886+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:18.893417+0000) 2019-10-18T08:03:18.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:03:18.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:18.922980+0000) 2019-10-18T08:03:19.726 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:19.717+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:19.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:19.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:19.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:19.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:20.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:20.917+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:20.926 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:20.917+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:21.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:21.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:21.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:21.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:22.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:22.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:22.965 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:22.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:23.169 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:03:23.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:23.166855+0000) 2019-10-18T08:03:23.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:23.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:23.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:23.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:24.727 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:24.717+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:24.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:24.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:24.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:24.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:25.899 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:25.889+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:25.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:25.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:26.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:03:26.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:26.140903+0000) 2019-10-18T08:03:26.941 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:26.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:26.942 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:26.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:27.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:27.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:27.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:27.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:28.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:03:28.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:28.893528+0000) 2019-10-18T08:03:28.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:03:28.917+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:28.923077+0000) 2019-10-18T08:03:28.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:28.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:28.979 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:28.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:29.728 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:29.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:29.994 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:29.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:29.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:29.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:31.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:31.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:31.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:31.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:32.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:32.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:32.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:32.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:32.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:32.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:32.964 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:32.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:33.170 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:03:33.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:33.167037+0000) 2019-10-18T08:03:34.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:34.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:34.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:34.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:34.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:34.721+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:34.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:34.957+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:34.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:34.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:35.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:35.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:35.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:35.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:36.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:03:36.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:36.141015+0000) 2019-10-18T08:03:36.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:36.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:36.963 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:36.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:37.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:37.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:37.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:37.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:38.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:03:38.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:38.893642+0000) 2019-10-18T08:03:38.925 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:03:38.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:38.923176+0000) 2019-10-18T08:03:38.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:38.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:38.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:38.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:39.730 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:39.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:40.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:40.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:40.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:40.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:40.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:40.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:40.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:40.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:42.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:42.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:42.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:42.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:43.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:43.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:43.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:43.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:43.170 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:03:43.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:43.167221+0000) 2019-10-18T08:03:43.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:43.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:43.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:43.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:44.731 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:44.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:45.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:45.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:45.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:45.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:46.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:46.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:46.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:46.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:46.143 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:03:46.136+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:46.141125+0000) 2019-10-18T08:03:46.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:46.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:46.978 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:46.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:48.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:48.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:48.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:48.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:48.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:03:48.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:48.893758+0000) 2019-10-18T08:03:48.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:03:48.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:48.923266+0000) 2019-10-18T08:03:48.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:48.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:48.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:48.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:49.732 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:49.725+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:49.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:49.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:49.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:49.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:51.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:51.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:51.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:52.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:52.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:52.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:52.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:53.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:53.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:53.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:53.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:53.170 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:03:53.164+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:53.167435+0000) 2019-10-18T08:03:54.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:54.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:54.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:54.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:54.733 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:54.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:03:55.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:55.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:55.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:55.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:56.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:56.064+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:56.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:56.064+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:56.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:03:56.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:56.141236+0000) 2019-10-18T08:03:57.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:57.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:57.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:57.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:58.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:58.104+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:58.111 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:58.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:58.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:03:58.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:58.893871+0000) 2019-10-18T08:03:58.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:03:58.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:03:58.923365+0000) 2019-10-18T08:03:59.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:59.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:03:59.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:03:59.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:03:59.734 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:03:59.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:00.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:00.116+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:00.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:00.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:01.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:01.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:01.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:01.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:02.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:02.112+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:02.118 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:02.112+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:03.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:03.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:03.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:03.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:03.170 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:04:03.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:03.167622+0000) 2019-10-18T08:04:04.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:04.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:04.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:04.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:04.735 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:04.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:05.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:05.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:05.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:05.128+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:06.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:06.088+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:06.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:06.088+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:06.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:04:06.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:06.141349+0000) 2019-10-18T08:04:07.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:07.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:07.092 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:07.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:08.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:08.136+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:08.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:08.136+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:08.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:04:08.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:08.893985+0000) 2019-10-18T08:04:08.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:04:08.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:08.923466+0000) 2019-10-18T08:04:09.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:09.128+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:09.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:09.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:09.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:09.728+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:10.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:10.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:10.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:10.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:11.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:11.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:11.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:11.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:12.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:12.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:12.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:12.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:13.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:13.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:13.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:13.044+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:13.170 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:04:13.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:13.167812+0000) 2019-10-18T08:04:14.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:14.084+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:14.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:14.084+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:14.736 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:14.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:15.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:15.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:15.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:15.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:16.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:16.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:16.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:16.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:16.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:04:16.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:16.141473+0000) 2019-10-18T08:04:17.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:17.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:17.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:17.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:18.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:18.140+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:18.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:18.140+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:18.896 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:04:18.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:18.894100+0000) 2019-10-18T08:04:18.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:04:18.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:18.923565+0000) 2019-10-18T08:04:19.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:19.160+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:19.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:19.164+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:19.740 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:19.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:20.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:20.164+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:20.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:20.168+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:21.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:21.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:21.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:21.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:22.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:22.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:22.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:22.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:23.170 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:04:23.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:23.167954+0000) 2019-10-18T08:04:23.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:23.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:23.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:23.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:24.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:24.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:24.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:24.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:24.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:24.732+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:25.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:25.268+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:25.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:25.268+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:26.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:04:26.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:26.141586+0000) 2019-10-18T08:04:26.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:26.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:26.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:26.252+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:27.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:27.252+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:27.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:27.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:28.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:28.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:28.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:28.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:28.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:04:28.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:28.894212+0000) 2019-10-18T08:04:28.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:04:28.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:28.923663+0000) 2019-10-18T08:04:29.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:29.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:29.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:29.224+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:29.738 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:29.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:30.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:30.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:30.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:30.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:31.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:31.299+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:31.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:31.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:32.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:32.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:32.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:32.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:33.171 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:04:33.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:33.168166+0000) 2019-10-18T08:04:33.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:33.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:33.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:33.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:34.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:34.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:34.258 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:34.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:34.739 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:34.731+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:35.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:35.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:35.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:35.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:36.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:04:36.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:36.141698+0000) 2019-10-18T08:04:36.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:36.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:36.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:36.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:37.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:37.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:37.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:37.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:38.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:38.219+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:38.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:38.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:38.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:04:38.891+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:38.894328+0000) 2019-10-18T08:04:38.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:04:38.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:38.923762+0000) 2019-10-18T08:04:39.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:39.267+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:39.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:39.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:39.741 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:39.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:40.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:40.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:40.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:40.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:41.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:41.311+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:41.319 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:41.311+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:42.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:42.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:42.283 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:42.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:43.171 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:04:43.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:43.168375+0000) 2019-10-18T08:04:43.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:43.251+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:43.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:43.251+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:44.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:44.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:44.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:44.223+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:44.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:44.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:45.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:45.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:45.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:45.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:46.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:04:46.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:46.141796+0000) 2019-10-18T08:04:46.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:46.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:46.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:46.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:47.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:47.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:47.267 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:47.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:48.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:48.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:48.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:48.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:48.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:04:48.891+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:48.894446+0000) 2019-10-18T08:04:48.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:04:48.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:48.923856+0000) 2019-10-18T08:04:49.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:49.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:49.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:49.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:49.742 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:49.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:50.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:50.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:50.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:50.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:51.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:51.275+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:51.282 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:51.275+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:52.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:52.247+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:52.256 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:52.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:53.171 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:04:53.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:53.168561+0000) 2019-10-18T08:04:53.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:53.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:53.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:53.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:54.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:54.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:54.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:54.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:54.743 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:54.735+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:04:55.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:55.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:55.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:55.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:56.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:04:56.135+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:56.141910+0000) 2019-10-18T08:04:56.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:56.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:56.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:56.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:57.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:57.303+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:57.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:57.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:58.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:58.343+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:58.352 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:58.343+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:58.897 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:04:58.891+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:58.894573+0000) 2019-10-18T08:04:58.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:04:58.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:04:58.923954+0000) 2019-10-18T08:04:59.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:59.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:04:59.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:04:59.347+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:04:59.744 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:04:59.739+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:00.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:00.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:00.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:00.383+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:01.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:01.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:01.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:01.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:02.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:02.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:02.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:02.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:03.171 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:05:03.163+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:03.168773+0000) 2019-10-18T08:05:03.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:03.411+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:03.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:03.411+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:04.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:04.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:04.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:04.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:04.745 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:04.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:05.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:05.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:05.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:05.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:06.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:05:06.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:06.142023+0000) 2019-10-18T08:05:06.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:06.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:06.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:06.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:07.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:07.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:07.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:07.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:08.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:08.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:08.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:08.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:08.897 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:05:08.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:08.894688+0000) 2019-10-18T08:05:08.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:05:08.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:08.924054+0000) 2019-10-18T08:05:09.418 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:09.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:09.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:09.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:09.746 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:09.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:10.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:10.410+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:10.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:10.410+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:11.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:11.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:11.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:11.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:12.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:12.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:12.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:12.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:13.171 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:05:13.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:13.168989+0000) 2019-10-18T08:05:13.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:13.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:13.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:13.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:14.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:14.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:14.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:14.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:14.747 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:14.738+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:15.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:15.350+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:15.360 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:15.350+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:16.144 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:05:16.134+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:16.142134+0000) 2019-10-18T08:05:16.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:16.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:16.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:16.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:17.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:17.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:17.368 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:17.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:18.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:18.326+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:18.334 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:18.326+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:18.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:05:18.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:18.894805+0000) 2019-10-18T08:05:18.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:05:18.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:18.924155+0000) 2019-10-18T08:05:19.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:19.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:19.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:19.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:19.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:19.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:20.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:20.346+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:20.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:20.346+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:21.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:21.390+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:21.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:21.390+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:22.448 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:22.438+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:22.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:22.438+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:23.171 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:05:23.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:23.169201+0000) 2019-10-18T08:05:23.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:23.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:23.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:23.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:24.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:24.398+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:24.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:24.398+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:24.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:24.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:25.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:25.434+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:25.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:25.434+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:26.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:05:26.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:26.142264+0000) 2019-10-18T08:05:26.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:26.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:26.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:26.422+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:27.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:27.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:27.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:27.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:28.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:28.342+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:28.353 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:28.342+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:28.897 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:05:28.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:28.894918+0000) 2019-10-18T08:05:28.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:05:28.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:28.924254+0000) 2019-10-18T08:05:29.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:29.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:29.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:29.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:29.749 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:29.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:30.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:30.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:30.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:30.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:31.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:31.298+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:31.308 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:31.298+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:32.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:32.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:32.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:32.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:33.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:05:33.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:33.169393+0000) 2019-10-18T08:05:33.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:33.274+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:33.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:33.274+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:34.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:34.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:34.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:34.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:34.750 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:34.742+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:35.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:35.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:35.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:35.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:36.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:05:36.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:36.142361+0000) 2019-10-18T08:05:36.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:36.270+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:36.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:36.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:37.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:37.302+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:37.317 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:37.302+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:38.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:38.254+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:38.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:38.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:38.897 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:05:38.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:38.895036+0000) 2019-10-18T08:05:38.926 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:05:38.918+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:38.924354+0000) 2019-10-18T08:05:39.224 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:39.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:39.214+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:39.751 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:39.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:40.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:40.221+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:40.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:40.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:41.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:41.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:41.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:41.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:42.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:42.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:42.206 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:42.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:43.172 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:05:43.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:43.169575+0000) 2019-10-18T08:05:43.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:43.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:43.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:43.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:44.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:44.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:44.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:44.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:44.752 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:44.741+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:45.257+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:45.271 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:45.261+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:46.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:05:46.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:46.142459+0000) 2019-10-18T08:05:46.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:46.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:46.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:46.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:47.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:47.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:47.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:47.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:48.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:48.141+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:48.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:48.141+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:48.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:05:48.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:48.895162+0000) 2019-10-18T08:05:48.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:05:48.917+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:48.924450+0000) 2019-10-18T08:05:49.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:49.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:49.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:49.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:49.753 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:49.745+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:50.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:50.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:50.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:50.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:51.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:51.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:51.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:52.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:52.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:52.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:52.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:53.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:53.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:53.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:53.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:53.172 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:05:53.162+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:53.169761+0000) 2019-10-18T08:05:54.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:54.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:54.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:54.057+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:54.754 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:54.745+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:05:55.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:55.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:55.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:55.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:56.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:55.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:56.004 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:55.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:56.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:05:56.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:56.142565+0000) 2019-10-18T08:05:57.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:56.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:57.009 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:57.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:58.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:58.021+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:58.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:58.021+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:58.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:05:58.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:58.895280+0000) 2019-10-18T08:05:58.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:05:58.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:05:58.924551+0000) 2019-10-18T08:05:59.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:59.025+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:05:59.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:05:59.025+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:05:59.755 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:05:59.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:00.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:00.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:00.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:00.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:01.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:01.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:01.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:01.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:02.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:02.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:02.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:02.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:02.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:02.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:02.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:02.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:03.172 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:06:03.166+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:03.169976+0000) 2019-10-18T08:06:03.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:03.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:03.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:03.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:04.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:04.749+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:04.987 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:04.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:04.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:04.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:05.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:05.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:05.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:05.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:06.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:06:06.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:06.142679+0000) 2019-10-18T08:06:06.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:06.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:06.914 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:06.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:07.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:07.929+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:07.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:07.929+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:08.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:08.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:08.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:08.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:08.897 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:06:08.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:08.895396+0000) 2019-10-18T08:06:08.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:06:08.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:08.924656+0000) 2019-10-18T08:06:09.756 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:09.753+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:09.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:09.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:09.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:09.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:10.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:10.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:10.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:10.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 588 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:11.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:11.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:11.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:11.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:12.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:12.949+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:12.954 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:12.949+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:13.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:06:13.170+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:13.170209+0000) 2019-10-18T08:06:13.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:13.921+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:13.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:13.921+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:14.757 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:14.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:14.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:14.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:14.930 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:14.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:15.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:15.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:15.910 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:15.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:16.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:06:16.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:16.142793+0000) 2019-10-18T08:06:16.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:16.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:16.866 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:16.860+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:17.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:17.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:17.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:17.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:18.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:06:18.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:18.895514+0000) 2019-10-18T08:06:18.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:18.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:18.921 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:18.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:18.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:06:18.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:18.924754+0000) 2019-10-18T08:06:19.758 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:19.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:19.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:19.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:19.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:19.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:20.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:20.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:20.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:20.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:21.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:21.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:21.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:21.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:22.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:22.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:22.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:22.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:23.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:06:23.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:23.170371+0000) 2019-10-18T08:06:23.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:23.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:23.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:23.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:24.746 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:24.740+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:24.747 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:24.740+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:24.759 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:24.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:25.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:25.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:25.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:25.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:26.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:06:26.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:26.142907+0000) 2019-10-18T08:06:26.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:26.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:26.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:26.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:27.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:27.720+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:27.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:27.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:28.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:28.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:28.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:06:28.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:28.895630+0000) 2019-10-18T08:06:28.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:06:28.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:28.924853+0000) 2019-10-18T08:06:29.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:29.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:29.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:29.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:29.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:29.752+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:30.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:30.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:30.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:30.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:31.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:31.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:31.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:31.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:32.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:32.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:32.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:32.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:33.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:06:33.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:33.170524+0000) 2019-10-18T08:06:33.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:33.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:33.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:33.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:34.760 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:34.756+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:34.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:34.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:34.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:34.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:35.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:35.768+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:35.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:35.768+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:36.145 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:06:36.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:36.143052+0000) 2019-10-18T08:06:36.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:36.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:36.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:36.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:37.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:37.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:37.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:37.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:38.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:38.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:38.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:38.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:38.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:06:38.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:38.895750+0000) 2019-10-18T08:06:38.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:06:38.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:38.924954+0000) 2019-10-18T08:06:39.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:39.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:39.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:39.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:39.762 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:39.756+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:40.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:40.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:40.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:40.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:41.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:41.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:41.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:41.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:42.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:42.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:42.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:42.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:43.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:06:43.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:43.170701+0000) 2019-10-18T08:06:43.758 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:43.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:43.759 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:43.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:44.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:44.736+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:44.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:44.736+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:44.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:44.756+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:45.702 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:45.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:45.703 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:45.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:46.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:06:46.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:46.143166+0000) 2019-10-18T08:06:46.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:46.712+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:46.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:46.712+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:47.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:47.684+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:47.693 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:47.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:48.642 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:48.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:48.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:48.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:48.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:06:48.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:48.895862+0000) 2019-10-18T08:06:48.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:06:48.920+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:48.925054+0000) 2019-10-18T08:06:49.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:49.611+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:49.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:49.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:49.763 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:49.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:50.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:50.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:50.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:50.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:51.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:51.551+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:51.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:51.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:52.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:52.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:52.581 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:52.575+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:53.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:06:53.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:53.170884+0000) 2019-10-18T08:06:53.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:53.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:53.569 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:53.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:54.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:54.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:54.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:54.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:54.764 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:54.755+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:06:55.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:55.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:55.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:55.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:56.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:06:56.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:56.143265+0000) 2019-10-18T08:06:56.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:56.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:56.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:56.619+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:57.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:57.631+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:57.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:57.631+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:58.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:58.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:58.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:58.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:58.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:06:58.891+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:58.895972+0000) 2019-10-18T08:06:58.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:06:58.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:06:58.925151+0000) 2019-10-18T08:06:59.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:59.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:06:59.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:06:59.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:06:59.765 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:06:59.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:00.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:00.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:00.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:00.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:01.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:01.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:01.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:01.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:02.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:02.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:02.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:03.173 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:03.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:03.171066+0000) 2019-10-18T08:07:03.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:03.667+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:03.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:03.667+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:04.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:04.683+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:04.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:04.683+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:04.766 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:04.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:05.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:05.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:05.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:05.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:06.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:07:06.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:06.143378+0000) 2019-10-18T08:07:06.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:06.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:06.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:06.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:07.686 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:07.679+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:07.687 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:07.679+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:08.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:08.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:08.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:08.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:08.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:07:08.891+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:08.896085+0000) 2019-10-18T08:07:08.927 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:07:08.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:08.925253+0000) 2019-10-18T08:07:09.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:09.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:09.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:09.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:09.767 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:09.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:10.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:10.719+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:10.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:10.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:11.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:11.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:11.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:11.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:12.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:12.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:12.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:12.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:13.177 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:13.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:13.171237+0000) 2019-10-18T08:07:13.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:13.747+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:13.761 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:13.747+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:14.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:14.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:14.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:14.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:14.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:14.759+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:15.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:15.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:15.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:15.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:16.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:07:16.141+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:16.143491+0000) 2019-10-18T08:07:16.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:16.711+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:16.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:16.715+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:17.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:17.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:17.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:17.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:18.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:18.771+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:18.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:18.771+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:18.898 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:07:18.891+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:18.896215+0000) 2019-10-18T08:07:18.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:07:18.919+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:18.925348+0000) 2019-10-18T08:07:19.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:19.751+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:19.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:19.751+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:19.769 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:19.763+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:20.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:20.743+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:20.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:20.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:21.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:21.707+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:21.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:21.707+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:22.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:22.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:22.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:22.691+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:23.174 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:23.169+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:23.171405+0000) 2019-10-18T08:07:23.175 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:23.169+0000 7f45251c4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:23.172260+0000) 2019-10-18T08:07:23.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:23.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:23.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:23.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:24.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:24.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:24.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:24.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:24.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:24.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:25.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:25.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:25.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:25.678+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:26.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:07:26.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:26.143608+0000) 2019-10-18T08:07:26.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:26.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:26.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:26.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:27.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:27.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:27.671824+0000) 2019-10-18T08:07:27.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:27.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:27.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:27.734+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:28.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:28.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:28.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:28.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:28.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:07:28.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:28.896326+0000) 2019-10-18T08:07:28.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:07:28.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:28.925457+0000) 2019-10-18T08:07:29.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:29.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:29.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:29.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:29.770 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:29.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:30.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:30.638+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:30.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:30.638+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:31.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:31.662+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:31.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:31.666+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:32.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:32.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:32.718 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:32.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:33.722+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:33.722+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:34.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:34.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:34.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:34.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:34.771 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:34.762+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:35.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:35.738+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:35.748 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:35.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:36.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:07:36.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:36.143722+0000) 2019-10-18T08:07:36.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:36.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:36.722 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:36.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:37.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:37.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:37.672002+0000) 2019-10-18T08:07:37.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:37.690+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:37.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:37.690+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:38.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:38.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:38.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:38.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:38.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:07:38.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:38.896443+0000) 2019-10-18T08:07:38.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:07:38.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:38.925556+0000) 2019-10-18T08:07:39.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:39.734+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:39.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:39.738+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:39.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:39.766+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:40.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:40.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:40.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:40.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:41.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:41.726+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:41.737 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:41.726+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:42.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:42.706+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:42.716 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:42.706+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:43.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:43.686+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:43.695 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:43.686+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:44.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:44.674+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:44.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:44.674+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:44.773 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:44.766+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:45.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:45.642+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:45.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:45.642+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:46.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:07:46.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:46.143843+0000) 2019-10-18T08:07:46.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:46.634+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:46.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:46.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:47.666+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:47.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:47.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:47.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:47.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:47.672185+0000) 2019-10-18T08:07:48.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:48.710+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:48.719 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:48.710+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:48.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:07:48.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:48.896557+0000) 2019-10-18T08:07:48.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:07:48.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:48.925653+0000) 2019-10-18T08:07:49.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:49.714+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:49.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:49.714+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:49.774 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:49.766+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:50.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:50.750+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T07:57:14.523367+0000 (oldest deadline 2019-10-18T07:57:34.523367+0000) 2019-10-18T08:07:50.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:50.750+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:51.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:51.786+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:52.837 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:52.826+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:53.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:53.846+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:54.775 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:54.766+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:54.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:54.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:55.830 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:55.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:56.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:07:56.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:56.143951+0000) 2019-10-18T08:07:56.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:56.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:57.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:07:57.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:57.672396+0000) 2019-10-18T08:07:57.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:57.822+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:58.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:58.798+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:07:58.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:07:58.890+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:58.896676+0000) 2019-10-18T08:07:58.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:07:58.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:07:58.925753+0000) 2019-10-18T08:07:59.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:07:59.765+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:07:59.828 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:07:59.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:00.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:00.789+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:01.812 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:01.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:02.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:02.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:03.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:03.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:04.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:04.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:04.776 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:04.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:05.742 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:05.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:06.146 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:08:06.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:06.144067+0000) 2019-10-18T08:08:06.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:06.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:07.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:08:07.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:07.672581+0000) 2019-10-18T08:08:07.757 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:07.749+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:08.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:08.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:08.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:08:08.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:08.896794+0000) 2019-10-18T08:08:08.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:08:08.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:08.925852+0000) 2019-10-18T08:08:09.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:09.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:09.778 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:09.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:10.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:10.681+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:11.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:11.669+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:12.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:12.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:13.682 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:13.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:14.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:14.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:14.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:14.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:14.779 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:14.769+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:15.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:15.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:15.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:15.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:16.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:08:16.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:16.144182+0000) 2019-10-18T08:08:16.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:16.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:16.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:16.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:17.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:08:17.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:17.672762+0000) 2019-10-18T08:08:17.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:17.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:17.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:17.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:18.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:18.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:18.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:18.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:18.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:08:18.889+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:18.896910+0000) 2019-10-18T08:08:18.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:08:18.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:18.925952+0000) 2019-10-18T08:08:19.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:19.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:19.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:19.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:19.780 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:19.773+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:20.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:20.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:20.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:20.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:21.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:21.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:21.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:21.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:22.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:22.741+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:22.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:22.741+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:23.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:23.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:23.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:23.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:24.744 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:24.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:24.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:24.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:24.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:24.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:25.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:25.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:25.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:25.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:26.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:08:26.140+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:26.144295+0000) 2019-10-18T08:08:26.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:26.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:26.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:26.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:27.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:08:27.668+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:27.672972+0000) 2019-10-18T08:08:27.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:27.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:27.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:27.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:28.724 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:28.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:28.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:28.721+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:28.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:08:28.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:28.897021+0000) 2019-10-18T08:08:28.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:08:28.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:28.926053+0000) 2019-10-18T08:08:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:29.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:29.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:29.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:29.781 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:29.777+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:30.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:30.733+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:30.745 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:30.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:31.774 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:31.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:31.775 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:31.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:32.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:32.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:32.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:32.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:33.784 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:33.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:33.785 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:33.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:34.782 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:34.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:34.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:34.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:34.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:34.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:35.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:35.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:35.839 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:35.832+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:36.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:08:36.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:36.144411+0000) 2019-10-18T08:08:36.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:36.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:36.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:36.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:37.675 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:08:37.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:37.673154+0000) 2019-10-18T08:08:37.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:37.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:37.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:37.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:38.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:38.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:38.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:38.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:38.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:08:38.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:38.897142+0000) 2019-10-18T08:08:38.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:08:38.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:38.926154+0000) 2019-10-18T08:08:39.783 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:39.776+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:39.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:39.812+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:39.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:39.812+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:40.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:40.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:40.788 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:40.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:41.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:41.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:41.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:41.824+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:42.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:42.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:42.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:42.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:43.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:43.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:43.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:43.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:44.784 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:44.780+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:44.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:44.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:44.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:44.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:45.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:45.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:45.894 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:45.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:46.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:08:46.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:46.144527+0000) 2019-10-18T08:08:46.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:46.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:46.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:46.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:47.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:08:47.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:47.673340+0000) 2019-10-18T08:08:47.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:47.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:47.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:47.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:48.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:48.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:48.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:48.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:48.899 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:08:48.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:48.897261+0000) 2019-10-18T08:08:48.928 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:08:48.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:48.926254+0000) 2019-10-18T08:08:49.785 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:49.780+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:49.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:49.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:49.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:49.872+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:50.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:50.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:50.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:50.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:51.900 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:51.892+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:51.901 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:51.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:52.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:52.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:52.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:52.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:53.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:53.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:53.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:53.956+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:54.786 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:54.780+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:54.980 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:54.972+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:54.981 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:54.972+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:55.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:55.932+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:55.940 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:55.932+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:56.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:08:56.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:56.144629+0000) 2019-10-18T08:08:56.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:56.920+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:56.928 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:56.920+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:57.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:08:57.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:57.673527+0000) 2019-10-18T08:08:57.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:57.928+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:57.936 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:57.928+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:58.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:08:58.892+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:58.897378+0000) 2019-10-18T08:08:58.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:08:58.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:08:58.926356+0000) 2019-10-18T08:08:58.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:58.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:58.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:58.944+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:08:59.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:08:59.780+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:08:59.918 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:59.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:08:59.919 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:08:59.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:00.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:00.860+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:00.870 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:00.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:01.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:01.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:01.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:01.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:02.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:02.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:02.913 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:02.908+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:03.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:03.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:03.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:03.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:04.787 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:04.780+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:04.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:04.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:04.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:04.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:05.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:05.964+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:05.972 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:05.964+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:06.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:09:06.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:06.144743+0000) 2019-10-18T08:09:06.944 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:06.936+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:06.946 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:06.936+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:07.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:09:07.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:07.673712+0000) 2019-10-18T08:09:07.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:07.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:07.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:07.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:08.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:09:08.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:08.897493+0000) 2019-10-18T08:09:08.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:09:08.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:08.926455+0000) 2019-10-18T08:09:09.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:09.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:09.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:09.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:09.788 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:09.779+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:10.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:10.007+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:10.017 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:10.007+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:11.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:11.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:11.069 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:11.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:12.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:12.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:12.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:12.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:13.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:13.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:13.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:13.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:14.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:14.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:14.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:14.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:14.789 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:14.783+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:15.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:15.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:15.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:15.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:16.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:16.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:16.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:16.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:16.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:09:16.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:16.144860+0000) 2019-10-18T08:09:17.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:17.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:17.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:17.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:17.676 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:09:17.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:17.673900+0000) 2019-10-18T08:09:18.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:18.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:18.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:18.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:18.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:09:18.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:18.897603+0000) 2019-10-18T08:09:18.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:09:18.923+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:18.926554+0000) 2019-10-18T08:09:19.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:19.223+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:19.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:19.227+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:19.791 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:19.783+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:20.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:20.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:20.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:20.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:21.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:21.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:21.213 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:21.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:22.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:22.187+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:22.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:22.187+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:23.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:23.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:23.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:23.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:24.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:24.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:24.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:24.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:24.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:24.783+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:25.251 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:25.243+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:25.252 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:25.243+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:26.147 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:09:26.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:26.145024+0000) 2019-10-18T08:09:26.300 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:26.291+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:26.301 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:26.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:27.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:27.339+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:27.346 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:27.339+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:27.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:09:27.667+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:27.674085+0000) 2019-10-18T08:09:28.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:28.383+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:28.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:28.387+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:28.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:09:28.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:28.897725+0000) 2019-10-18T08:09:28.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:09:28.923+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:28.926655+0000) 2019-10-18T08:09:29.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:29.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:29.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:29.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:29.792 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:29.783+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:30.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:30.403+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:30.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:30.403+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:31.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:31.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:31.409 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:31.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:32.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:32.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:32.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:32.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:33.386 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:33.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:33.387 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:33.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:34.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:34.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:34.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:34.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:34.793 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:34.787+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:35.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:35.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:35.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:35.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:36.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:09:36.139+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:36.145240+0000) 2019-10-18T08:09:36.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:36.323+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:36.332 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:36.323+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:37.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:37.335+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:37.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:37.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:37.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:09:37.671+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:37.674261+0000) 2019-10-18T08:09:38.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:38.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:38.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:38.371+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:38.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:09:38.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:38.897848+0000) 2019-10-18T08:09:38.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:09:38.923+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:38.926754+0000) 2019-10-18T08:09:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:39.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:39.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:39.794 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:39.787+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:40.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:40.419+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:40.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:40.419+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:41.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:41.455+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:41.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:41.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:42.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:42.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:42.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:42.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:43.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:43.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:43.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:43.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:44.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:44.487+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:44.496 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:44.487+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:44.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:44.786+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:45.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:45.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:45.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:45.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:46.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:09:46.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:46.145420+0000) 2019-10-18T08:09:46.489 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:46.482+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:46.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:46.482+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:47.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:47.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:47.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:47.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:47.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:09:47.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:47.674437+0000) 2019-10-18T08:09:48.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:48.486+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:48.497 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:48.490+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:48.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:09:48.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:48.897964+0000) 2019-10-18T08:09:48.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:09:48.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:48.926857+0000) 2019-10-18T08:09:49.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:49.498+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:49.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:49.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:49.795 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:49.786+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:50.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:50.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:50.485 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:50.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:51.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:51.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:51.475 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:51.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:52.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:52.458+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:52.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:52.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:53.494+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:53.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:53.494+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:54.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:54.526+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:54.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:54.526+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:54.797 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:54.790+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:09:55.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:55.514+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:55.523 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:55.514+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:56.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:09:56.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:56.145602+0000) 2019-10-18T08:09:56.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:56.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:56.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:56.550+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:57.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:57.534+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:57.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:57.534+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:57.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:09:57.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:57.674584+0000) 2019-10-18T08:09:58.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:58.570+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:58.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:58.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:58.900 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:09:58.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:58.898081+0000) 2019-10-18T08:09:58.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:09:58.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:09:58.926953+0000) 2019-10-18T08:09:59.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:59.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:09:59.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:09:59.566+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:09:59.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:09:59.790+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:00.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:00.602+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:00.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:00.602+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:01.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:01.590+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:01.599 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:01.590+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:02.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:02.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:02.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:02.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:03.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:03.610+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:03.619 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:03.610+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:04.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:04.578+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:04.587 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:04.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:04.798 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:04.790+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:05.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:05.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:05.629 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:05.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:06.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:10:06.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:06.145791+0000) 2019-10-18T08:10:06.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:06.618+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:06.627 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:06.618+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:07.590 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:07.582+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:07.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:07.582+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:07.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:10:07.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:07.674756+0000) 2019-10-18T08:10:08.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:08.598+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:08.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:08.598+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:08.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:10:08.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:08.898196+0000) 2019-10-18T08:10:08.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:10:08.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:08.927050+0000) 2019-10-18T08:10:09.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:09.566+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:09.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:09.570+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:09.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:09.790+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:10.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:10.606+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:10.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:10.606+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:11.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:11.630+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:11.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:11.634+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:12.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:12.658+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:12.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:12.658+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:13.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:13.614+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:13.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:13.614+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:14.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:14.650+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:14.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:14.650+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:14.799 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:14.790+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:15.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:15.670+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:15.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:15.670+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:16.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:10:16.138+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:16.145990+0000) 2019-10-18T08:10:16.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:16.718+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:16.727 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:16.718+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:17.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:10:17.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:17.674951+0000) 2019-10-18T08:10:17.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:17.730+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:17.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:17.730+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:18.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:18.702+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:18.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:18.702+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:18.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:10:18.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:18.898311+0000) 2019-10-18T08:10:18.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:10:18.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:18.927147+0000) 2019-10-18T08:10:19.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:19.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:19.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:19.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:19.801 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:19.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:20.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:20.721+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:20.734 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:20.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:21.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:21.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:21.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:21.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:22.666 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:22.657+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:22.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:22.657+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:23.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:23.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:23.712 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:23.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:24.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:24.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:24.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:24.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:24.802 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:24.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:25.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:25.729+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:25.741 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:25.733+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:26.148 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:10:26.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:26.146198+0000) 2019-10-18T08:10:26.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:26.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:26.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:26.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:27.677 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:10:27.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:27.675130+0000) 2019-10-18T08:10:27.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:27.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:27.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:27.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:28.699 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:28.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:28.700 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:28.689+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:28.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:10:28.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:28.898431+0000) 2019-10-18T08:10:28.929 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:10:28.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:28.927247+0000) 2019-10-18T08:10:29.658 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:29.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:29.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:29.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:29.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:29.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:30.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:30.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:30.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:30.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:31.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:31.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:31.583 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:31.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:32.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:32.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:32.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:32.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:33.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:33.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:33.628 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:33.617+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:34.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:34.665+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:34.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:34.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:34.804 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:34.793+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:35.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:35.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:35.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:35.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:36.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:10:36.142+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:36.146372+0000) 2019-10-18T08:10:36.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:36.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:36.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:36.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:37.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:10:37.670+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:37.675314+0000) 2019-10-18T08:10:37.688 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:37.677+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:37.689 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:37.677+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:38.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:38.701+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:38.713 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:38.701+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:38.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:10:38.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:38.898553+0000) 2019-10-18T08:10:38.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:10:38.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:38.927367+0000) 2019-10-18T08:10:39.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:39.713+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:39.720 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:39.713+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:39.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:39.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:40.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:40.693+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:40.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:40.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:41.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:41.689+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:41.698 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:41.693+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:42.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:42.685+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:42.692 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:42.685+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:43.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:43.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:43.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:43.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:44.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:44.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:44.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:44.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:44.805 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:44.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:45.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:45.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:45.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:45.633+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:46.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:10:46.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:46.146551+0000) 2019-10-18T08:10:46.614 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:46.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:46.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:46.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:47.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:47.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:47.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:47.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:47.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:10:47.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:47.675523+0000) 2019-10-18T08:10:48.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:48.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:48.669 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:48.665+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:48.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:10:48.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:48.898672+0000) 2019-10-18T08:10:48.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:10:48.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:48.927466+0000) 2019-10-18T08:10:49.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:49.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:49.631 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:49.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:49.806 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:49.801+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:50.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:50.609+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:50.615 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:50.609+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:51.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:51.617+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:51.625 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:51.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:52.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:52.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:52.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:52.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:53.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:53.637+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:53.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:53.641+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:54.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:54.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:54.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:54.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:54.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:54.800+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:10:55.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:55.608+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:55.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:55.608+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:56.149 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:10:56.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:56.146739+0000) 2019-10-18T08:10:56.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:56.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:56.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:56.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:57.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:57.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:57.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:57.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:57.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:10:57.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:57.675704+0000) 2019-10-18T08:10:58.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:58.612+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:58.618 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:58.612+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:58.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:10:58.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:58.898793+0000) 2019-10-18T08:10:58.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:10:58.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:10:58.927564+0000) 2019-10-18T08:10:59.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:59.568+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:10:59.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:10:59.568+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:10:59.807 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:10:59.800+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:00.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:00.604+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:00.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:00.604+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:01.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:01.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:01.641 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:01.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:02.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:02.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:02.643 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:02.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:03.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:03.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:03.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:03.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:04.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:04.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:04.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:04.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:04.809 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:04.804+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:05.696 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:05.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:05.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:05.692+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:06.162 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:11:06.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:06.146951+0000) 2019-10-18T08:11:06.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:06.692+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:06.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:06.696+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:07.678 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:11:07.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:07.675908+0000) 2019-10-18T08:11:07.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:07.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:07.680 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:07.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:08.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:08.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:08.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:08.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:08.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:11:08.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:08.898912+0000) 2019-10-18T08:11:08.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:11:08.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:08.927663+0000) 2019-10-18T08:11:09.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:09.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:09.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:09.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:09.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:09.804+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:10.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:10.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:10.715 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:10.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:11.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:11.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:11.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:11.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:12.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:12.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:12.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:12.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:13.755 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:13.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:13.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:13.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:14.780 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:14.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:14.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:14.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:14.810 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:14.804+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:15.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:15.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:15.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:15.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:16.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:11:16.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:16.147142+0000) 2019-10-18T08:11:16.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:16.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:16.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:16.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:17.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:11:17.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:17.676119+0000) 2019-10-18T08:11:17.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:17.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:17.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:17.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:18.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:18.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:18.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:18.901 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:11:18.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:18.899031+0000) 2019-10-18T08:11:18.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:11:18.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:18.927759+0000) 2019-10-18T08:11:19.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:19.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:19.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:19.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:19.811 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:19.804+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:20.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:20.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:20.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:20.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:21.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:21.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:21.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:21.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:22.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:22.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:22.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:22.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:23.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:23.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:23.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:23.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:24.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:24.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:24.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:24.796+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:24.812 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:24.808+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:25.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:25.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:25.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:25.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:26.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:11:26.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:26.147356+0000) 2019-10-18T08:11:26.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:26.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:26.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:26.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:27.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:11:27.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:27.676328+0000) 2019-10-18T08:11:27.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:27.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:27.847 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:27.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:28.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:28.832+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:28.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:28.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:28.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:11:28.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:28.899152+0000) 2019-10-18T08:11:28.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:11:28.924+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:28.927855+0000) 2019-10-18T08:11:29.813 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:29.807+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:29.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:29.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:29.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:29.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:30.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:30.763+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:30.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:30.763+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:31.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:31.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:31.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:31.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:32.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:32.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:32.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:32.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:33.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:33.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:33.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:33.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:34.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:34.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:34.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:34.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:34.814 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:34.807+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:35.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:35.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:35.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:35.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:36.150 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:11:36.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:36.147536+0000) 2019-10-18T08:11:36.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:36.775+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:36.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:36.775+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:37.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:11:37.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:37.676518+0000) 2019-10-18T08:11:37.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:37.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:37.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:37.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:38.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:38.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:38.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:38.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:38.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:11:38.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:38.899273+0000) 2019-10-18T08:11:38.930 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:11:38.923+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:38.927954+0000) 2019-10-18T08:11:39.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:39.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:39.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:39.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:39.815 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:39.807+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:40.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:40.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:40.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:40.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:41.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:41.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:41.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:41.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:42.783+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:42.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:42.783+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:43.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:43.755+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:43.762 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:43.755+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:44.804 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:44.795+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:44.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:44.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:44.816 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:44.811+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:45.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:45.827+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:45.836 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:45.827+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:46.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:11:46.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:46.147733+0000) 2019-10-18T08:11:46.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:46.871+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:46.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:46.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:47.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:11:47.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:47.676733+0000) 2019-10-18T08:11:47.884 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:47.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:47.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:47.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:48.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:11:48.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:48.899392+0000) 2019-10-18T08:11:48.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:48.923+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:48.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:48.923+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:48.931 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:11:48.923+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:48.928583+0000) 2019-10-18T08:11:49.817 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:49.811+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:49.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:49.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:49.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:49.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:50.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:50.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:50.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:50.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:51.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:51.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:51.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:51.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:53.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:53.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:53.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:53.011+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:54.067 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:54.059+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:54.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:54.059+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:54.818 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:54.811+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:11:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:55.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:55.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:55.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:56.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:56.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:56.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:56.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:56.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:11:56.145+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:56.147947+0000) 2019-10-18T08:11:57.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:57.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:57.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:57.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:57.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:11:57.673+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:57.676945+0000) 2019-10-18T08:11:58.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:58.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:58.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:58.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:58.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:11:58.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:58.899528+0000) 2019-10-18T08:11:58.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:11:58.923+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:11:58.928684+0000) 2019-10-18T08:11:59.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:59.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:11:59.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:11:59.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:11:59.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:11:59.811+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:00.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:00.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:00.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:00.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:01.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:01.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:01.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:01.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:02.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:02.107+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:02.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:02.107+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:03.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:03.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:03.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:03.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:04.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:04.091+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:04.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:04.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:04.819 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:04.810+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:05.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:05.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:05.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:05.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:06.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:06.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:06.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:06.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:06.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:12:06.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:06.148123+0000) 2019-10-18T08:12:07.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:07.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:07.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:07.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:07.679 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:12:07.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:07.677138+0000) 2019-10-18T08:12:08.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:08.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:08.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:08.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:08.902 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:12:08.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:08.899642+0000) 2019-10-18T08:12:08.931 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:12:08.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:08.928790+0000) 2019-10-18T08:12:09.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:09.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:09.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:09.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:09.820 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:09.810+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:10.033 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:10.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:10.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:10.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:10.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:10.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:10.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:10.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:12.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:12.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:12.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:12.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:13.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:13.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:13.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:13.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:14.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:14.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:14.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:14.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:14.821 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:14.814+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:15.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:15.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:15.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:15.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:16.116 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:16.106+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:16.117 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:16.106+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:16.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:12:16.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:16.148314+0000) 2019-10-18T08:12:17.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:17.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:17.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:17.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:17.680 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:12:17.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:17.677352+0000) 2019-10-18T08:12:18.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:18.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:18.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:18.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:18.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:12:18.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:18.899756+0000) 2019-10-18T08:12:18.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:12:18.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:18.928890+0000) 2019-10-18T08:12:19.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:19.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:19.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:19.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:19.822 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:19.814+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:20.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:20.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:20.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:20.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:21.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:21.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:21.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:21.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:22.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:22.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:22.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:22.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:23.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:23.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:23.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:23.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:24.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:24.198+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:24.209 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:24.198+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:24.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:24.814+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:25.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:25.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:25.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:25.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:26.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:12:26.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:26.148526+0000) 2019-10-18T08:12:26.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:26.202+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:26.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:26.202+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:27.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:27.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:27.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:27.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:27.680 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:12:27.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:27.677520+0000) 2019-10-18T08:12:28.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:28.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:28.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:28.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:28.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:12:28.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:28.899875+0000) 2019-10-18T08:12:28.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:12:28.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:28.928987+0000) 2019-10-18T08:12:29.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:29.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:29.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:29.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:29.823 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:29.814+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:30.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:30.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:30.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:30.162+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:31.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:31.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:31.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:31.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:32.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:32.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:32.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:32.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:33.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:33.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:33.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:33.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:34.159 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:34.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:34.160 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:34.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:34.824 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:34.818+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:35.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:35.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:35.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:35.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:36.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:12:36.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:36.148708+0000) 2019-10-18T08:12:36.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:36.222+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:36.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:36.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:37.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:37.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:37.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:37.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:37.680 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:12:37.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:37.677722+0000) 2019-10-18T08:12:38.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:38.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:38.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:38.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:38.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:12:38.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:38.899991+0000) 2019-10-18T08:12:38.931 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:12:38.922+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:38.929087+0000) 2019-10-18T08:12:39.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:39.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:39.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:39.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:39.825 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:39.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:40.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:40.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:40.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:40.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:41.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:41.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:41.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:41.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:42.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:42.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:42.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:42.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:43.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:43.053+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:43.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:43.053+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:44.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:44.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:44.091 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:44.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:44.826 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:44.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:45.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:45.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:45.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:45.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:46.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:46.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:46.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:46.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:46.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:12:46.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:46.148891+0000) 2019-10-18T08:12:47.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:47.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:47.127 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:47.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:47.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:12:47.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:47.677930+0000) 2019-10-18T08:12:48.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:48.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:48.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:48.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:48.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:12:48.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:48.900105+0000) 2019-10-18T08:12:48.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:12:48.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:48.929183+0000) 2019-10-18T08:12:49.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:49.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:49.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:49.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:49.827 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:49.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:50.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:50.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:50.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:50.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:51.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:51.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:51.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:51.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:52.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:52.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:52.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:52.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:53.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:53.145+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:53.156 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:53.145+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:54.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:54.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:54.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:54.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:54.828 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:54.817+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:12:55.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:55.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:55.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:55.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:56.151 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:12:56.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:56.149071+0000) 2019-10-18T08:12:56.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:56.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:56.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:56.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:57.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:57.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:57.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:57.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:57.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:12:57.672+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:57.678114+0000) 2019-10-18T08:12:58.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:58.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:58.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:58.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:58.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:12:58.893+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:58.900220+0000) 2019-10-18T08:12:58.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:12:58.921+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:12:58.929282+0000) 2019-10-18T08:12:59.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:59.189+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:12:59.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:12:59.189+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:12:59.829 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:12:59.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:00.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:00.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:00.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:00.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:01.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:01.209+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:01.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:01.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:02.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:02.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:02.243 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:02.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:03.230 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:03.225+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:03.231 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:03.225+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:04.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:04.237+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:04.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:04.237+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:04.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:04.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:05.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:05.233+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:05.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:05.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:06.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:13:06.144+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:06.149254+0000) 2019-10-18T08:13:06.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:06.265+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:06.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:06.265+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:07.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:07.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:07.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:07.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:07.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:13:07.676+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:07.678318+0000) 2019-10-18T08:13:08.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:08.313+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:08.321 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:08.317+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:08.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:13:08.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:08.900349+0000) 2019-10-18T08:13:08.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:13:08.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:08.929380+0000) 2019-10-18T08:13:09.311 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:09.305+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:09.312 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:09.305+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:09.830 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:09.825+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:10.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:10.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:10.339 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:10.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:11.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:11.353+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:11.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:11.353+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:12.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:12.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:12.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:12.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:13.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:13.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:13.344 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:13.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:14.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:14.385+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:14.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:14.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:14.831 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:14.824+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:15.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:15.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:15.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:15.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:16.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:13:16.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:16.149435+0000) 2019-10-18T08:13:16.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:16.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:16.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:16.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:17.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:17.424+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:17.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:17.424+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:17.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:13:17.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:17.678466+0000) 2019-10-18T08:13:18.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:18.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:18.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:18.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:18.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:13:18.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:18.900462+0000) 2019-10-18T08:13:18.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:13:18.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:18.929465+0000) 2019-10-18T08:13:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:19.456+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:19.467 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:19.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:19.832 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:19.828+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:20.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:20.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:20.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:20.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:21.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:21.496+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:21.504 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:21.496+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:22.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:22.512+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:22.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:22.512+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:23.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:23.556+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:23.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:23.556+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:24.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:24.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:24.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:24.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:24.834 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:24.828+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:25.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:25.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:25.498 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:25.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:26.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:13:26.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:26.149626+0000) 2019-10-18T08:13:26.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:26.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:26.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:26.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:27.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:27.540+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:27.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:27.540+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:27.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:13:27.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:27.678647+0000) 2019-10-18T08:13:28.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:28.504+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:28.513 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:28.504+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:28.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:13:28.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:28.900584+0000) 2019-10-18T08:13:28.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:13:28.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:28.929561+0000) 2019-10-18T08:13:29.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:29.492+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:29.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:29.492+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:29.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:29.828+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:30.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:30.464+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:30.470 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:30.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:31.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:31.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:31.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:31.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:32.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:32.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:32.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:32.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:33.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:33.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:33.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:33.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:34.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:34.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:34.490 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:34.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:34.835 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:34.828+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:35.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:35.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:35.509 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:35.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:36.152 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:13:36.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:36.149824+0000) 2019-10-18T08:13:36.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:36.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:36.473 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:36.464+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:37.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:37.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:37.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:37.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:37.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:13:37.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:37.678849+0000) 2019-10-18T08:13:38.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:38.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:38.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:38.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:38.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:13:38.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:38.900704+0000) 2019-10-18T08:13:38.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:13:38.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:38.929658+0000) 2019-10-18T08:13:39.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:39.396+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:39.402 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:39.396+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:39.836 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:39.828+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:40.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:40.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:40.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:40.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:41.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:41.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:41.461 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:41.456+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:42.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:42.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:42.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:42.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:43.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:43.532+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:43.538 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:43.532+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:44.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:44.500+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:44.507 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:44.500+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:44.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:44.832+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:45.529 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:45.524+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:45.530 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:45.524+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:46.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:13:46.143+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:46.150002+0000) 2019-10-18T08:13:46.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:46.488+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:46.494 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:46.488+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:47.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:47.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:47.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:47.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:47.681 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:13:47.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:47.679029+0000) 2019-10-18T08:13:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:48.564+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:48.564+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:48.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:13:48.896+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:48.900821+0000) 2019-10-18T08:13:48.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:13:48.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:48.929752+0000) 2019-10-18T08:13:49.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:49.536+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:49.542 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:49.536+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:49.837 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:49.831+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:50.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:50.523+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:50.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:50.523+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:51.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:51.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:51.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:51.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:52.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:52.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:52.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:52.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:53.556 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:53.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:53.557 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:53.551+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:54.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:54.599+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:54.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:54.599+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:54.838 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:54.831+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:13:55.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:55.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:55.591 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:55.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:56.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:13:56.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:56.150169+0000) 2019-10-18T08:13:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:56.623+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:56.633 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:56.627+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:57.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:57.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:57.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:57.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:57.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:13:57.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:57.679251+0000) 2019-10-18T08:13:58.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:58.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:58.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:58.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:58.919 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:13:58.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:58.900940+0000) 2019-10-18T08:13:58.932 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:13:58.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:13:58.929850+0000) 2019-10-18T08:13:59.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:59.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:13:59.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:13:59.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:13:59.840 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:13:59.831+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:00.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:00.579+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:00.589 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:00.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:01.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:01.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:01.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:01.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:02.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:02.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:02.624 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:02.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:03.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:03.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:03.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:03.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:04.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:04.595+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:04.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:04.595+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:04.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:04.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:05.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:05.555+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:05.562 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:05.555+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:06.160 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:14:06.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:06.150383+0000) 2019-10-18T08:14:06.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:06.559+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:06.567 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:06.559+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:07.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:07.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:07.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:07.607+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:07.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:14:07.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:07.679436+0000) 2019-10-18T08:14:08.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:08.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:08.580 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:08.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:08.903 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:14:08.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:08.901058+0000) 2019-10-18T08:14:08.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:14:08.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:08.929949+0000) 2019-10-18T08:14:09.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:09.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:09.612 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:09.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:09.841 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:09.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:10.616 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:10.607+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:10.617 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:10.611+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:11.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:11.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:11.577 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:11.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:12.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:12.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:12.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:12.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:13.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:13.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:13.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:13.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:14.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:14.591+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:14.598 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:14.591+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:14.842 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:14.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:15.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:15.571+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:15.579 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:15.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:16.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:14:16.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:16.150571+0000) 2019-10-18T08:14:16.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:16.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:16.578 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:16.571+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:17.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:17.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:17.593 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:17.587+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:17.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:14:17.675+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:17.679625+0000) 2019-10-18T08:14:18.551 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:18.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:18.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:18.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:18.906 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:14:18.895+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:18.901196+0000) 2019-10-18T08:14:18.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:14:18.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:18.930046+0000) 2019-10-18T08:14:19.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:19.539+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:19.549 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:19.539+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:19.843 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:19.835+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:20.502 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:20.495+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:20.503 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:20.495+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:21.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:21.511+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:21.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:21.511+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:22.552 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:22.543+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:22.553 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:22.543+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:23.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:23.583+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:23.592 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:23.583+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:24.574 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:24.567+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:24.575 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:24.567+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:24.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:24.834+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:25.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:25.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:25.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:25.574+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:26.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:14:26.147+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:26.150756+0000) 2019-10-18T08:14:26.606 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:26.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:26.607 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:26.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:27.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:27.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:27.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:27.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:27.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:14:27.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:27.679804+0000) 2019-10-18T08:14:28.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:28.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:28.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:28.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:28.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:14:28.894+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:28.901326+0000) 2019-10-18T08:14:28.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:14:28.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:28.930143+0000) 2019-10-18T08:14:29.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:29.558+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:29.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:29.558+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:29.844 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:29.838+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:30.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:30.574+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:30.585 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:30.578+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:31.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:31.538+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:31.548 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:31.538+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:32.533 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:32.518+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:32.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:32.518+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:33.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:33.502+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:33.510 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:33.502+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:34.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:34.478+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:34.488 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:34.478+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:34.845 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:34.838+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:35.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:35.454+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:35.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:35.458+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:36.153 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:14:36.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:36.150934+0000) 2019-10-18T08:14:36.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:36.422+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:36.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:36.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:37.426 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:37.418+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:37.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:37.418+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:37.682 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:14:37.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:37.679982+0000) 2019-10-18T08:14:38.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:38.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:38.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:38.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:38.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:14:38.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:38.901454+0000) 2019-10-18T08:14:38.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:14:38.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:38.930237+0000) 2019-10-18T08:14:39.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:39.466+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:39.474 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:39.466+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:39.847 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:39.838+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:40.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:40.462+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:40.471 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:40.462+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:41.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:41.442+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:41.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:41.442+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:42.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:42.470+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:42.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:42.470+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:43.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:43.446+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:43.456 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:43.446+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:44.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:44.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:44.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:44.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:44.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:44.838+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:45.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:45.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:45.396 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:45.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:46.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:14:46.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:46.151114+0000) 2019-10-18T08:14:46.422 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:46.414+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:46.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:46.414+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:47.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:47.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:47.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:47.378+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:47.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:14:47.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:47.680166+0000) 2019-10-18T08:14:48.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:48.386+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:48.394 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:48.386+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:48.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:14:48.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:48.901575+0000) 2019-10-18T08:14:48.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:14:48.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:48.930330+0000) 2019-10-18T08:14:49.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:49.354+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:49.365 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:49.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:49.848 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:49.838+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:50.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:50.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:50.328 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:50.318+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:51.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:51.358+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:51.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:51.358+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:52.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:52.378+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:52.389 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:52.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:53.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:53.426+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:53.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:53.426+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:54.441 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:54.430+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:54.442 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:54.430+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:54.849 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:54.842+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:14:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:55.474+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:55.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:55.474+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:56.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:14:56.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:56.151298+0000) 2019-10-18T08:14:56.520 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:56.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:56.521 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:56.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:57.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:57.510+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:57.518 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:57.510+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:57.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:14:57.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:57.680352+0000) 2019-10-18T08:14:58.560 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:58.550+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:58.561 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:58.554+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:58.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:14:58.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:58.901694+0000) 2019-10-18T08:14:58.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:14:58.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:14:58.930427+0000) 2019-10-18T08:14:59.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:59.594+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:14:59.602 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:14:59.594+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:14:59.850 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:14:59.841+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:00.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:00.577+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:00.588 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:00.577+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:01.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:01.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:01.632 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:01.621+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:02.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:02.573+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:02.584 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:02.573+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:03.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:03.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:03.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:03.533+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:04.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:04.533+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:04.546 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:04.537+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:04.850 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:04.841+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:05.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:05.497+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:05.508 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:05.497+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:06.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:06.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:06.151472+0000) 2019-10-18T08:15:06.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:06.525+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:06.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:06.525+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:07.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:07.517+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:07.527 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:07.517+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:07.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:15:07.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:07.680537+0000) 2019-10-18T08:15:08.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:08.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:08.566 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:08.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:08.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:15:08.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:08.901809+0000) 2019-10-18T08:15:08.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:15:08.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:08.930526+0000) 2019-10-18T08:15:09.608 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:09.597+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:09.609 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:09.597+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:09.852 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:09.841+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:10.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:10.629+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:10.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:10.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:11.654 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:11.645+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:11.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:11.645+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:12.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:12.633+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:12.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:12.637+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:13.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:13.661+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:13.672 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:13.661+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:14.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:14.649+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:14.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:14.649+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:14.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:14.845+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:15.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:15.625+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:15.638 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:15.629+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:16.154 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:16.146+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:16.151643+0000) 2019-10-18T08:15:16.155 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:16.146+0000 7f5520f0a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:16.152602+0000) 2019-10-18T08:15:16.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:16.621+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:16.634 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:16.625+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:17.681 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:17.673+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:17.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:15:17.674+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:17.680721+0000) 2019-10-18T08:15:17.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:17.673+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:18.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:18.697+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:18.706 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:18.697+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:18.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:15:18.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:18.901924+0000) 2019-10-18T08:15:18.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:15:18.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:18.930621+0000) 2019-10-18T08:15:19.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:19.717+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:19.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:19.717+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:19.853 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:19.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:20.655 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:20.646+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:20.652068+0000) 2019-10-18T08:15:20.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:20.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:20.773 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:20.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:21.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:21.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:21.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:21.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:22.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:22.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:22.802 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:22.793+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:23.789 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:23.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:23.790 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:23.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:24.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:24.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:24.766 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:24.761+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:24.854 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:24.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:25.730 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:25.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:25.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:25.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:26.769 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:26.761+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:26.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:26.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:27.683 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:15:27.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:27.680906+0000) 2019-10-18T08:15:27.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:27.737+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:27.743 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:27.737+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:28.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:28.785+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:28.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:28.785+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:28.904 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:15:28.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:28.902038+0000) 2019-10-18T08:15:28.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:15:28.929+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:28.930717+0000) 2019-10-18T08:15:29.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:29.821+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:29.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:29.821+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:29.855 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:29.849+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:30.655 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:30.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:30.652256+0000) 2019-10-18T08:15:30.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:30.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:30.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:30.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:31.765+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:31.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:31.765+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:32.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:32.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:32.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:32.725+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:33.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:33.725+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:33.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:33.729+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:34.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:34.769+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:34.778 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:34.769+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:34.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:34.848+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:35.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:35.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:35.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:35.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:36.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:36.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:36.803 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:36.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:37.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:15:37.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:37.681057+0000) 2019-10-18T08:15:37.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:37.836+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:37.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:37.836+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:38.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:38.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:38.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:38.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:38.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:15:38.900+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:38.902155+0000) 2019-10-18T08:15:38.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:15:38.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:38.930814+0000) 2019-10-18T08:15:39.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:39.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:39.820 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:39.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:39.856 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:39.852+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:40.655 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:40.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:40.652441+0000) 2019-10-18T08:15:40.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:40.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:40.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:40.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:41.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:41.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:41.754 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:41.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:42.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:42.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:42.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:42.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:43.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:43.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:43.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:43.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:44.791 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:44.784+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:44.792 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:44.784+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:44.857 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:44.852+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:45.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:45.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:45.772 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:45.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:46.795 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:46.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:46.796 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:46.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:47.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:15:47.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:47.681240+0000) 2019-10-18T08:15:47.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:47.800+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:47.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:47.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:48.822 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:48.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:48.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:48.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:48.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:15:48.900+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:48.902275+0000) 2019-10-18T08:15:48.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:15:48.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:48.930910+0000) 2019-10-18T08:15:49.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:49.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:49.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:49.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:49.859 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:49.852+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:50.655 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:15:50.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:50.652661+0000) 2019-10-18T08:15:50.826 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:50.820+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:50.827 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:50.820+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:51.805 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:51.796+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:51.806 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:51.800+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:52.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:52.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:52.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:52.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:53.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:53.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:53.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:53.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:54.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:54.772+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:54.779 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:54.772+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:54.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:54.852+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:15:55.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:55.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:55.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:55.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:56.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:56.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:56.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:56.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:57.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:15:57.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:57.681424+0000) 2019-10-18T08:15:57.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:57.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:57.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:57.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:58.732 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:58.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:58.733 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:58.728+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:58.907 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:15:58.900+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:58.902395+0000) 2019-10-18T08:15:58.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:15:58.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:15:58.931007+0000) 2019-10-18T08:15:59.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:59.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:15:59.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:15:59.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:15:59.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:15:59.852+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:00.655 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:16:00.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:00.652850+0000) 2019-10-18T08:16:00.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:00.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:00.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:00.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:01.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:01.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:01.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:01.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:02.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:02.804+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:02.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:02.804+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:03.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:03.848+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:03.858 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:03.852+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:04.851 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:04.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:04.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:04.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:04.860 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:04.856+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:05.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:05.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:05.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:05.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:06.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:06.900+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:06.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:06.900+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:07.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:16:07.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:07.681611+0000) 2019-10-18T08:16:07.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:07.880+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:07.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:07.880+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:08.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:08.868+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:08.877 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:08.868+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:08.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:16:08.900+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:08.902512+0000) 2019-10-18T08:16:08.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:16:08.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:08.931105+0000) 2019-10-18T08:16:09.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:09.840+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:09.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:09.840+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:09.861 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:09.855+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:10.657 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:16:10.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:10.653063+0000) 2019-10-18T08:16:10.864 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:10.855+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:10.865 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:10.855+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:11.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:11.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:11.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:11.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:12.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:12.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:12.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:12.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:13.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:13.843+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:13.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:13.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:14.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:14.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:14.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:14.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:14.862 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:14.855+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:15.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:15.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:15.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:15.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:16.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:16.883+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:16.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:16.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:17.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:16:17.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:17.681821+0000) 2019-10-18T08:16:17.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:17.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:17.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:17.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:18.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:16:18.899+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:18.902630+0000) 2019-10-18T08:16:18.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:18.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:18.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:18.899+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:18.933 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:16:18.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:18.931200+0000) 2019-10-18T08:16:19.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:19.855+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:19.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:19.935+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:19.943 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:19.935+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:20.656 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:16:20.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:20.653246+0000) 2019-10-18T08:16:20.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:20.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:20.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:20.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:21.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:21.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:21.968 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:21.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:22.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:22.959+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:22.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:22.959+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:24.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:24.003+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:24.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:24.003+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:24.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:24.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:25.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:25.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:25.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:25.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:25.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:25.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:25.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:25.967+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:26.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:26.975+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:26.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:26.975+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:27.684 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:16:27.677+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:27.682034+0000) 2019-10-18T08:16:28.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:28.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:28.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:28.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:28.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:16:28.899+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:28.902747+0000) 2019-10-18T08:16:28.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:16:28.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:28.931294+0000) 2019-10-18T08:16:29.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:29.019+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:29.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:29.019+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:29.865 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:29.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:30.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:30.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:30.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:30.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:30.657 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:16:30.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:30.653432+0000) 2019-10-18T08:16:31.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:31.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:31.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:31.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:32.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:32.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:32.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:32.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:33.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:33.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:33.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:33.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:34.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:34.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:34.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:34.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:34.866 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:34.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:35.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:35.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:35.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:35.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:36.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:36.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:36.103 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:36.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:37.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:37.143+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:37.151 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:37.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:37.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:16:37.681+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:37.682233+0000) 2019-10-18T08:16:38.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:38.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:38.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:38.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:38.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:16:38.899+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:38.902865+0000) 2019-10-18T08:16:38.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:16:38.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:38.931390+0000) 2019-10-18T08:16:39.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:39.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:39.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:39.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:39.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:39.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:40.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:40.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:40.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:40.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:40.661 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:16:40.649+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:40.653595+0000) 2019-10-18T08:16:41.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:41.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:41.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:41.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:42.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:42.079+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:42.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:42.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:43.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:43.103+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:43.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:43.103+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:44.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:44.075+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:44.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:44.079+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:44.867 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:44.859+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:45.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:45.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:45.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:45.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:46.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:46.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:46.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:46.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:47.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:47.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:47.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:47.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:47.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:16:47.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:47.682379+0000) 2019-10-18T08:16:48.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:48.022+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:48.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:48.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:48.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:16:48.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:48.902981+0000) 2019-10-18T08:16:48.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:16:48.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:48.931488+0000) 2019-10-18T08:16:49.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:49.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:49.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:49.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:49.869 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:49.858+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:50.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:50.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:50.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:50.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:50.656 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:16:50.648+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:50.653779+0000) 2019-10-18T08:16:51.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:51.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:51.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:51.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:52.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:52.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:52.077 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:52.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:53.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:53.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:53.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:53.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:54.054 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:54.046+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:54.055 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:54.046+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:54.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:54.862+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:16:55.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:55.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:55.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:55.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:56.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:56.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:56.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:56.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:57.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:57.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:57.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:57.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:57.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:16:57.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:57.682534+0000) 2019-10-18T08:16:58.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:58.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:58.082 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:58.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:58.905 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:16:58.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:58.903102+0000) 2019-10-18T08:16:58.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:16:58.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:16:58.931586+0000) 2019-10-18T08:16:59.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:59.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:16:59.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:16:59.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:16:59.870 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:16:59.862+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:00.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:00.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:00.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:00.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:00.661 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:17:00.648+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:00.653970+0000) 2019-10-18T08:17:01.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:01.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:01.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:01.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:02.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:02.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:02.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:02.130+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:03.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:03.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:03.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:03.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:04.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:04.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:04.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:04.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:04.871 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:04.862+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:05.070+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:05.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:05.070+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:06.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:06.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:06.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:06.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:07.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:07.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:07.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:07.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:07.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:17:07.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:07.682705+0000) 2019-10-18T08:17:08.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:08.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:08.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:08.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:08.906 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:08.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:08.903219+0000) 2019-10-18T08:17:08.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:17:08.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:08.931685+0000) 2019-10-18T08:17:09.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:09.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:09.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:09.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:09.872 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:09.862+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:10.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:10.238+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:10.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:10.238+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:10.657 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:17:10.652+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:10.654163+0000) 2019-10-18T08:17:11.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:11.226+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:11.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:11.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:12.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:12.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:12.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:12.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:13.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:13.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:13.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:13.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:14.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:14.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:14.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:14.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:14.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:14.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:15.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:15.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:15.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:15.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:16.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:16.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:16.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:16.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:17.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:17.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:17.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:17.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:17.685 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:17:17.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:17.682925+0000) 2019-10-18T08:17:18.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:18.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:18.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:18.174+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:18.906 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:18.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:18.903336+0000) 2019-10-18T08:17:18.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:17:18.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:18.931783+0000) 2019-10-18T08:17:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:19.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:19.155 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:19.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:19.873 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:19.866+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:20.157+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:20.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:20.157+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:20.657 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:17:20.652+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:20.654376+0000) 2019-10-18T08:17:21.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:21.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:21.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:21.117+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:22.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:22.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:22.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:22.093+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:23.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:23.081+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:23.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:23.081+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:24.112 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:24.101+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:24.113 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:24.101+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:24.874 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:24.865+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:25.072 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:25.061+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:25.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:25.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:26.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:26.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:26.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:26.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:27.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:27.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:27.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:27.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:27.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:17:27.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:27.683112+0000) 2019-10-18T08:17:28.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:28.077+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:28.089 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:28.077+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:28.906 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:28.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:28.903445+0000) 2019-10-18T08:17:28.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:17:28.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:28.931883+0000) 2019-10-18T08:17:29.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:29.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:29.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:29.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:29.875 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:29.865+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:30.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:30.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:30.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:30.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:30.658 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:17:30.652+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:30.654560+0000) 2019-10-18T08:17:31.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:31.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:31.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:31.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:32.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:32.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:32.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:32.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:33.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:33.033+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:33.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:33.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:34.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:34.085+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:34.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:34.085+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:34.876 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:34.865+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:35.128 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:35.117+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:35.129 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:35.121+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:36.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:36.133+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:36.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:36.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:37.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:37.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:37.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:37.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:37.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:17:37.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:37.683301+0000) 2019-10-18T08:17:38.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:38.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:38.179 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:38.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:38.906 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:38.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:38.903556+0000) 2019-10-18T08:17:38.934 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:17:38.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:38.931982+0000) 2019-10-18T08:17:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:39.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:39.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:39.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:39.877 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:39.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:40.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:40.229+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:40.238 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:40.233+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:40.657 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:17:40.652+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:40.654726+0000) 2019-10-18T08:17:41.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:41.217+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:41.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:41.217+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:42.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:42.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:42.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:42.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:43.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:43.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:43.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:43.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:44.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:44.197+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:44.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:44.197+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:44.878 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:44.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:45.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:45.181+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:45.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:45.181+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:46.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:46.213+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:46.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:46.213+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:47.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:47.169+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:47.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:47.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:47.686 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:17:47.680+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:47.683511+0000) 2019-10-18T08:17:48.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:48.177+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:48.184 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:48.177+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:48.906 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:48.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:48.903665+0000) 2019-10-18T08:17:48.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:48.901+0000 7fb64232d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:48.904847+0000) 2019-10-18T08:17:48.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:17:48.929+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:48.932081+0000) 2019-10-18T08:17:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:49.129+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:49.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:49.133+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:49.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:49.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:50.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:50.149+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:50.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:50.149+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:50.664 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:17:50.652+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:50.654912+0000) 2019-10-18T08:17:51.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:51.161+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:51.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:51.161+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:52.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:52.201+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:52.236 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:52.205+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:53.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:53.205+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:53.215 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:53.209+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:53.406 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:17:53.401+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:53.403844+0000) 2019-10-18T08:17:54.199 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:54.193+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:54.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:54.193+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:54.879 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:54.873+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:17:55.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:55.180+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:55.187 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:55.180+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:56.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:56.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:56.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:56.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:57.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:57.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:57.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:17:57.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:57.683724+0000) 2019-10-18T08:17:58.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:58.156+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:58.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:58.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:58.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:17:58.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:17:58.932183+0000) 2019-10-18T08:17:59.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:59.204+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:17:59.210 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:17:59.204+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:17:59.880 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:17:59.872+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:00.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:00.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:00.253 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:00.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:00.658 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:18:00.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:00.655080+0000) 2019-10-18T08:18:01.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:01.284+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:01.292 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:01.284+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:02.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:02.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:02.255 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:02.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:03.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:03.280+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:03.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:03.280+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:03.407 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:18:03.400+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:03.403945+0000) 2019-10-18T08:18:04.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:04.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:04.310 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:04.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:04.881 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:04.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:05.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:05.272+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:05.279 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:05.272+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:06.306 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:06.300+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:06.307 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:06.300+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:07.263 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:07.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:07.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:07.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:07.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:18:07.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:07.683914+0000) 2019-10-18T08:18:08.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:08.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:08.249 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:08.244+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:08.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:18:08.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:08.932287+0000) 2019-10-18T08:18:09.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:09.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:09.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:09.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:09.882 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:09.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:10.232 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:10.224+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:10.233 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:10.228+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:10.658 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:18:10.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:10.655268+0000) 2019-10-18T08:18:11.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:11.256+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:11.262 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:11.256+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:12.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:12.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:12.244 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:12.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:13.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:13.188+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:13.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:13.188+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:13.406 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:18:13.400+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:13.404043+0000) 2019-10-18T08:18:14.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:14.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:14.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:14.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:14.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:14.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:15.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:15.236+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:15.242 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:15.236+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:16.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:16.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:16.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:16.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:17.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:17.248+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:17.254 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:17.248+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:17.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:18:17.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:17.684120+0000) 2019-10-18T08:18:18.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:18.200+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:18.208 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:18.200+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:18.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:18:18.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:18.932390+0000) 2019-10-18T08:18:19.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:19.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:19.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:19.212+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:19.883 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:19.876+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:20.246 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:20.240+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:20.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:20.240+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:20.658 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:18:20.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:20.655454+0000) 2019-10-18T08:18:21.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:21.216+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:21.225 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:21.220+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:22.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:22.208+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:22.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:22.208+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:23.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:23.232+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:23.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:23.232+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:23.406 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:18:23.400+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:23.404138+0000) 2019-10-18T08:18:24.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:24.212+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:24.221 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:24.216+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:24.885 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:24.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:25.202 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:25.196+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:25.203 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:25.196+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:26.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:26.172+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:07:54.661694+0000 (oldest deadline 2019-10-18T08:08:14.661694+0000) 2019-10-18T08:18:26.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:26.172+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:27.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:27.184+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:27.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:18:27.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:27.684302+0000) 2019-10-18T08:18:28.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:28.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:28.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:18:28.928+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:28.932496+0000) 2019-10-18T08:18:29.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:29.116+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:29.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:29.880+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:30.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:30.139+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:30.662 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:18:30.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:30.655636+0000) 2019-10-18T08:18:31.197 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:31.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:32.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:32.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:33.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:33.211+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:33.408 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:18:33.399+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:33.404234+0000) 2019-10-18T08:18:34.227 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:34.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:34.886 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:34.879+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:35.278 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:35.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:36.257 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:36.247+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:37.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:37.291+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:37.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:18:37.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:37.684494+0000) 2019-10-18T08:18:38.284 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:38.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:38.939 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:18:38.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:38.932600+0000) 2019-10-18T08:18:39.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:39.303+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:39.887 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:39.879+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:40.290 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:40.283+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:40.658 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:18:40.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:40.655832+0000) 2019-10-18T08:18:41.326 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:41.319+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:42.313 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:42.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:43.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:43.267+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:43.407 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:18:43.399+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:43.404336+0000) 2019-10-18T08:18:44.286 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:44.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:44.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:44.879+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:45.309 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:45.299+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:46.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:46.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:47.342 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:47.335+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:47.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:18:47.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:47.684694+0000) 2019-10-18T08:18:48.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:48.363+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:48.370 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:48.363+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:48.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:18:48.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:48.932701+0000) 2019-10-18T08:18:49.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:49.367+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:49.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:49.367+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:49.888 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:49.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:50.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:50.355+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:50.366 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:50.359+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:50.659 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:18:50.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:50.656025+0000) 2019-10-18T08:18:51.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:51.379+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:51.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:51.379+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:52.430 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:52.423+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:52.431 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:52.423+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:53.407 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:18:53.399+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:53.404438+0000) 2019-10-18T08:18:53.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:53.467+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:53.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:53.467+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:54.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:54.471+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:54.479 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:54.471+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:54.890 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:54.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:18:55.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:55.515+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:55.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:55.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:56.536 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:56.527+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:56.537 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:56.527+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:57.544 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:57.531+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:57.545 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:57.535+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:57.687 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:18:57.679+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:57.684877+0000) 2019-10-18T08:18:58.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:58.547+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:58.565 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:58.547+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:58.935 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:18:58.927+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:18:58.932802+0000) 2019-10-18T08:18:59.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:59.563+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:18:59.571 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:18:59.563+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:18:59.892 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:18:59.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:00.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:00.519+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:00.528 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:00.519+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:00.659 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:19:00.651+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:00.656212+0000) 2019-10-18T08:19:01.500 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:01.491+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:01.501 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:01.491+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:02.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:02.459+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:02.466 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:02.459+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:03.407 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:03.399+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:03.404533+0000) 2019-10-18T08:19:03.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:03.439+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:03.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:03.443+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:04.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:04.399+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:04.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:04.399+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:04.891 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:04.883+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:05.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:05.394+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:05.406 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:05.394+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:06.390 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:06.382+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:06.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:06.382+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:07.412 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:07.402+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:07.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:07.406+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:07.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:19:07.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:07.685066+0000) 2019-10-18T08:19:08.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:08.366+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:08.376 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:08.366+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:08.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:19:08.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:08.932903+0000) 2019-10-18T08:19:09.329 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:09.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:09.330 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:09.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:09.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:09.882+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:10.295 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:10.286+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:10.296 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:10.286+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:10.659 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:19:10.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:10.656397+0000) 2019-10-18T08:19:11.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:11.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:11.259 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:11.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:12.298 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:12.290+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:12.299 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:12.290+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:13.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:13.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:13.260 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:13.250+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:13.407 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:13.398+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:13.404631+0000) 2019-10-18T08:19:14.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:14.250+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:14.261 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:14.254+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:14.893 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:14.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:15.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:15.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:15.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:15.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:16.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:16.178+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:16.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:16.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:17.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:17.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:17.154 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:17.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:17.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:19:17.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:17.685278+0000) 2019-10-18T08:19:18.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:18.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:18.196 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:18.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:18.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:19:18.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:18.933004+0000) 2019-10-18T08:19:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:19.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:19.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:19.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:19.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:19.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:20.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:20.162+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:20.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:20.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:20.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:19:20.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:20.656576+0000) 2019-10-18T08:19:21.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:21.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:21.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:21.146+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:22.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:22.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:22.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:22.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:23.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:23.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:23.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:23.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:23.407 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:23.398+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:23.404729+0000) 2019-10-18T08:19:23.408 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:23.402+0000 7fb642b2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:23.405568+0000) 2019-10-18T08:19:24.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:24.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:24.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:24.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:24.894 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:24.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:25.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:25.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:25.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:25.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:26.051 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:26.042+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:26.052 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:26.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:27.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:27.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:27.014 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:27.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:27.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:19:27.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:27.685463+0000) 2019-10-18T08:19:27.907 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:27.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:27.904923+0000) 2019-10-18T08:19:28.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:28.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:28.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:28.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:28.945 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:19:28.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:28.933109+0000) 2019-10-18T08:19:29.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:29.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:29.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:29.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:29.895 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:29.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:30.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:30.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:30.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:30.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:30.659 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:19:30.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:30.656776+0000) 2019-10-18T08:19:31.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:31.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:31.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:31.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:32.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:32.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:32.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:32.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:33.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:33.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:33.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:33.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:33.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:33.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:33.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:33.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:34.896 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:34.886+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:35.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:35.006+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:35.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:35.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:36.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:36.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:36.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:36.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:37.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:37.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:37.074 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:37.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:37.688 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:19:37.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:37.685653+0000) 2019-10-18T08:19:37.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:37.898+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:37.905022+0000) 2019-10-18T08:19:38.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:38.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:38.087 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:38.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:38.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:19:38.926+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:38.933212+0000) 2019-10-18T08:19:39.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:39.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:39.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:39.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:39.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:39.890+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:40.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:40.065+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:40.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:40.065+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:40.659 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:19:40.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:40.656971+0000) 2019-10-18T08:19:41.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:41.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:41.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:41.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:42.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:42.005+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:42.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:42.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:42.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:42.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:42.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:42.989+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:43.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:43.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:43.955 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:43.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:44.897 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:44.889+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:44.989 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:44.981+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:44.990 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:44.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:45.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:45.945+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:45.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:45.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:46.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:46.973+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:46.984 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:46.973+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:47.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:19:47.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:47.685840+0000) 2019-10-18T08:19:47.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:47.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:47.905117+0000) 2019-10-18T08:19:47.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:47.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:47.952 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:47.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:48.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:48.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:48.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:48.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:48.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:19:48.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:48.933319+0000) 2019-10-18T08:19:49.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:49.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:49.860 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:49.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:49.899 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:49.889+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:50.660 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:19:50.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:50.657140+0000) 2019-10-18T08:19:50.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:50.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:50.857 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:50.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:51.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:51.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:51.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:51.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:52.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:52.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:52.886 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:52.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:53.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:53.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:53.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:53.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:54.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:54.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:54.814 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:54.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:54.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:54.889+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:19:55.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:55.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:55.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:55.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:56.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:56.845+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:56.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:56.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:57.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:19:57.678+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:57.686027+0000) 2019-10-18T08:19:57.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:57.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:57.848 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:57.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:57.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:19:57.897+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:57.905215+0000) 2019-10-18T08:19:58.868 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:58.857+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:58.869 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:58.857+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:58.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:19:58.925+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:19:58.933425+0000) 2019-10-18T08:19:59.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:59.873+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:19:59.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:19:59.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:19:59.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:19:59.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:00.660 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:20:00.650+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:00.657341+0000) 2019-10-18T08:20:00.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:00.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:00.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:00.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:01.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:01.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:01.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:01.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:02.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:02.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:02.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:02.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:03.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:03.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:03.835 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:03.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:04.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:04.817+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:04.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:04.817+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:04.901 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:04.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:05.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:05.773+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:05.781 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:05.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:06.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:06.781+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:06.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:06.781+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:07.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:20:07.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:07.686231+0000) 2019-10-18T08:20:07.782 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:07.777+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:07.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:07.777+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:07.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:20:07.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:07.905315+0000) 2019-10-18T08:20:08.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:08.805+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:08.811 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:08.805+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:08.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:20:08.933+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:08.933566+0000) 2019-10-18T08:20:09.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:09.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:09.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:09.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:09.902 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:09.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:10.660 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:20:10.654+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:10.657538+0000) 2019-10-18T08:20:10.817 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:10.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:10.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:10.813+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:11.831 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:11.825+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:11.832 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:11.825+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:12.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:12.793+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:12.801 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:12.797+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:13.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:13.801+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:13.808 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:13.801+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:14.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:14.757+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:14.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:14.757+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:14.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:14.897+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:15.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:15.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:15.763 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:15.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:16.752 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:16.744+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:16.753 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:16.744+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:17.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:20:17.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:17.686372+0000) 2019-10-18T08:20:17.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:17.716+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:17.728 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:17.716+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:17.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:20:17.900+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:17.905414+0000) 2019-10-18T08:20:18.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:18.676+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:18.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:18.676+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:18.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:20:18.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:18.933669+0000) 2019-10-18T08:20:19.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:19.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:19.661 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:19.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:19.903 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:19.896+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:20.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:20.636+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:20.644 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:20.636+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:20.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:20:20.653+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:20.657712+0000) 2019-10-18T08:20:21.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:21.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:21.653 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:21.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:22.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:22.656+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:22.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:22.656+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:23.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:23.648+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:23.655 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:23.648+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:24.704 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:24.696+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:24.705 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:24.700+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:24.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:24.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:25.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:25.708+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:25.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:25.708+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:26.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:26.732+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:26.740 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:26.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:27.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:20:27.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:27.686538+0000) 2019-10-18T08:20:27.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:27.752+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:27.760 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:27.752+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:27.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:20:27.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:27.905522+0000) 2019-10-18T08:20:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:28.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:28.714 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:28.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:28.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:20:28.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:28.933770+0000) 2019-10-18T08:20:29.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:29.724+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:29.731 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:29.724+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:29.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:29.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:30.669 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:20:30.653+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:30.657925+0000) 2019-10-18T08:20:30.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:30.748+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:30.756 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:30.748+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:31.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:31.704+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:31.711 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:31.704+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:32.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:32.688+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:32.697 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:32.688+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:33.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:33.728+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:33.739 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:33.732+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:34.690 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:34.680+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:34.691 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:34.684+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:34.906 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:34.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:35.683 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:35.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:35.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:35.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:36.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:36.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:36.639 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:36.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:37.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:37.632+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:37.640 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:37.632+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:37.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:20:37.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:37.686712+0000) 2019-10-18T08:20:37.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:20:37.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:37.905623+0000) 2019-10-18T08:20:38.659 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:38.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:38.660 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:38.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:38.936 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:20:38.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:38.933870+0000) 2019-10-18T08:20:39.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:39.628+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:39.635 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:39.628+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:39.907 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:39.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:40.661 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:20:40.653+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:40.658139+0000) 2019-10-18T08:20:40.676 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:40.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:40.677 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:40.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:41.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:41.672+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:41.679 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:41.672+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:42.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:42.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:42.675 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:42.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:43.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:43.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:43.646 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:43.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:44.640+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:44.647 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:44.640+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:44.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:44.900+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:45.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:45.652+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:45.662 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:45.652+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:46.673 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:46.668+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:46.674 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:46.668+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:47.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:47.660+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:47.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:47.660+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:47.689 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:20:47.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:47.686894+0000) 2019-10-18T08:20:47.908 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:20:47.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:47.905723+0000) 2019-10-18T08:20:48.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:48.664+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:48.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:48.664+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:48.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:20:48.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:48.933967+0000) 2019-10-18T08:20:49.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:49.644+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:49.652 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:49.644+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:49.908 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:49.904+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:50.622 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:50.615+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:50.623 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:50.615+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:50.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:20:50.657+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:50.658328+0000) 2019-10-18T08:20:51.648 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:51.639+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:51.649 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:51.639+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:52.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:52.603+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:52.610 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:52.603+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:53.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:53.647+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:53.656 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:53.647+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:54.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:54.663+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:54.671 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:54.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:54.910 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:54.903+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:20:55.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:55.671+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:55.678 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:55.671+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:56.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:56.687+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:56.694 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:56.687+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:57.668 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:57.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:57.670 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:57.663+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:57.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:20:57.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:57.687082+0000) 2019-10-18T08:20:57.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:20:57.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:57.905820+0000) 2019-10-18T08:20:58.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:58.691+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:58.701 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:58.695+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:58.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:20:58.931+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:20:58.934068+0000) 2019-10-18T08:20:59.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:59.659+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:20:59.667 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:20:59.659+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:20:59.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:20:59.903+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:00.661 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:21:00.657+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:00.658525+0000) 2019-10-18T08:21:00.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:00.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:00.685 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:00.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:01.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:01.675+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:01.684 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:01.675+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:02.735 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:02.723+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:02.736 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:02.723+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:03.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:03.731+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:03.738 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:03.731+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:04.725 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:04.715+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:04.726 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:04.719+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:04.911 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:04.903+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:05.749 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:05.739+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:05.750 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:05.743+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:06.776 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:06.767+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:06.777 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:06.767+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:07.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:21:07.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:07.687267+0000) 2019-10-18T08:21:07.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:07.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:07.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:07.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:07.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:21:07.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:07.905920+0000) 2019-10-18T08:21:08.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:08.807+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:08.819 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:08.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:08.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:21:08.931+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:08.934169+0000) 2019-10-18T08:21:09.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:09.831+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:09.841 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:09.831+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:09.912 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:09.903+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:10.661 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:21:10.657+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:10.658739+0000) 2019-10-18T08:21:10.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:10.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:10.813 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:10.807+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 600 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:11.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:11.791+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:11.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:11.791+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:12.843 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:12.835+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:12.844 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:12.835+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:13.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:13.787+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:13.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:13.787+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:14.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:14.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:14.807 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:14.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:14.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:14.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:15.809 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:15.799+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:15.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:15.799+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:16.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:16.815+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:16.825 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:16.815+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:17.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:21:17.685+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:17.687462+0000) 2019-10-18T08:21:17.849 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:17.839+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:17.850 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:17.843+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:17.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:21:17.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:17.906015+0000) 2019-10-18T08:21:18.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:18.803+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:18.810 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:18.803+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:18.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:21:18.931+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:18.934271+0000) 2019-10-18T08:21:19.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:19.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:19.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:19.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:19.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:19.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:20.662 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:21:20.657+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:20.658881+0000) 2019-10-18T08:21:20.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:20.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:20.887 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:20.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:21.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:21.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:21.885 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:21.879+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:22.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:22.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:22.925 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:22.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:23.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:23.903+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:23.917 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:23.907+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:24.914 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:24.907+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:24.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:24.915+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:24.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:24.915+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:25.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:25.910+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:25.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:25.910+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:26.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:26.958+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:26.969 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:26.962+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:27.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:21:27.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:27.687668+0000) 2019-10-18T08:21:27.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:21:27.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:27.906110+0000) 2019-10-18T08:21:28.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:28.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:28.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:28.006+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:28.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:21:28.930+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:28.934372+0000) 2019-10-18T08:21:29.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:29.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:29.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:29.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:29.915 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:29.906+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:30.062 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:30.054+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:30.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:30.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:30.662 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:21:30.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:30.659070+0000) 2019-10-18T08:21:31.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:31.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:31.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:31.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:32.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:32.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:32.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:32.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:33.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:33.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:33.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:33.014+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:34.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:34.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:34.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:34.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:34.916 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:34.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:35.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:35.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:35.061 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:35.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:36.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:36.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:36.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:36.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:37.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:37.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:37.063 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:37.054+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:37.690 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:21:37.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:37.687869+0000) 2019-10-18T08:21:37.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:21:37.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:37.906208+0000) 2019-10-18T08:21:38.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:38.066+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:38.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:38.066+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:38.937 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:21:38.930+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:38.934474+0000) 2019-10-18T08:21:39.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:39.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:39.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:39.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:39.917 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:39.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:40.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:40.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:40.046 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:40.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:40.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:21:40.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:40.659283+0000) 2019-10-18T08:21:41.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:41.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:41.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:41.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:42.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:42.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:42.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:43.124 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:43.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:43.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:43.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:44.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:44.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:44.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:44.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:44.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:44.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:45.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:45.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:45.073 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:45.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:46.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:46.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:46.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:46.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:47.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:47.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:47.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:47.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:47.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:21:47.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:47.688049+0000) 2019-10-18T08:21:47.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:21:47.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:47.906306+0000) 2019-10-18T08:21:48.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:48.058+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:48.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:48.058+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:48.953 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:21:48.930+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:48.934579+0000) 2019-10-18T08:21:49.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:49.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:49.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:49.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:49.918 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:49.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:50.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:50.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:50.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:50.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:50.662 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:21:50.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:50.659452+0000) 2019-10-18T08:21:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:51.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:51.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:51.022+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:52.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:52.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:52.020 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:52.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:52.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:52.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:52.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:52.986+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:54.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:54.018+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:54.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:54.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:54.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:54.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:21:55.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:54.990+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:55.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:54.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:56.044 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:56.034+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:56.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:56.034+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:57.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:57.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:57.045 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:57.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:57.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:21:57.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:57.688244+0000) 2019-10-18T08:21:57.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:21:57.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:57.906401+0000) 2019-10-18T08:21:58.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:58.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:58.083 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:58.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:58.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:21:58.930+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:21:58.934679+0000) 2019-10-18T08:21:59.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:59.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:21:59.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:21:59.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:21:59.920 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:21:59.910+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:00.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:00.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:00.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:00.026+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:00.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:22:00.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:00.659603+0000) 2019-10-18T08:22:01.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:01.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:01.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:01.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:02.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:02.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:02.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:02.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:03.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:03.069+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:03.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:03.069+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:04.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:04.057+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:04.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:04.061+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:04.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:04.913+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:05.100 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:05.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:05.101 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:05.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:06.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:06.125+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:06.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:06.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:07.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:07.165+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:07.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:07.169+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:07.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:22:07.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:07.688434+0000) 2019-10-18T08:22:07.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:22:07.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:07.906500+0000) 2019-10-18T08:22:08.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:08.185+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:08.194 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:08.185+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:08.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:08.929+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:08.934783+0000) 2019-10-18T08:22:09.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:09.137+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:09.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:09.137+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:09.922 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:09.913+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:10.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:10.121+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:10.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:10.125+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:10.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:22:10.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:10.659771+0000) 2019-10-18T08:22:11.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:11.089+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:11.099 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:11.089+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:12.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:12.073+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:12.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:12.073+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:13.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:13.037+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:13.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:13.037+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:14.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:14.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:14.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:14.029+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:14.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:14.913+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:15.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:15.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:15.029 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:15.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:16.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:16.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:16.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:16.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:17.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:17.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:17.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:17.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:17.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:22:17.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:17.688619+0000) 2019-10-18T08:22:17.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:22:17.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:17.906597+0000) 2019-10-18T08:22:17.970 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:17.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:17.971 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:17.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:18.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:18.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:18.935 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:18.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:18.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:18.929+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:18.934887+0000) 2019-10-18T08:22:19.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:19.905+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:19.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:19.905+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:19.923 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:19.917+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:20.667 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:22:20.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:20.659991+0000) 2019-10-18T08:22:20.949 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:20.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:20.950 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:20.945+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:21.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:21.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:21.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:21.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:23.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:22.997+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:23.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:22.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:23.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:23.989+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:23.997 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:23.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:24.924 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:24.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:25.006 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:25.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:25.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:25.001+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:26.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:26.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:26.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:26.045+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:27.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:27.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:27.016 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:27.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:27.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:22:27.684+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:27.688776+0000) 2019-10-18T08:22:27.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:22:27.905+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:27.906692+0000) 2019-10-18T08:22:28.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:28.017+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:28.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:28.017+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:28.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:28.933+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:28.934990+0000) 2019-10-18T08:22:28.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:28.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:28.982 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:28.977+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:29.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:29.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:29.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:29.961+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:29.967 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:29.961+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:30.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:22:30.656+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:30.660205+0000) 2019-10-18T08:22:30.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:30.937+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:30.973 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:30.937+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:31.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:31.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:31.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:31.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:32.925+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:32.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:32.925+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:33.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:33.976 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:33.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:34.925 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:34.921+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:35.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:35.013+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:35.025 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:35.013+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:36.042 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:36.036+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:36.043 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:36.036+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:37.031 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:37.024+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:37.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:37.024+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:37.691 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:22:37.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:37.688928+0000) 2019-10-18T08:22:37.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:22:37.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:37.906792+0000) 2019-10-18T08:22:38.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:38.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:38.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:38.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:38.942 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:38.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:38.935096+0000) 2019-10-18T08:22:39.056 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:39.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:39.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:39.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:39.926 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:39.920+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:40.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:40.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:40.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:40.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:40.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:22:40.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:40.660349+0000) 2019-10-18T08:22:41.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:41.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:41.110 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:41.104+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:42.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:42.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:42.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:42.152+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:43.140 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:43.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:43.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:43.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:44.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:44.144+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:44.153 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:44.144+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:44.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:44.920+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:45.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:45.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:45.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:45.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:46.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:46.072+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:46.078 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:46.072+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:47.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:47.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:47.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:47.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:47.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:22:47.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:47.689108+0000) 2019-10-18T08:22:47.909 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:22:47.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:47.906891+0000) 2019-10-18T08:22:48.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:48.080+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:48.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:48.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:48.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:48.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:48.935202+0000) 2019-10-18T08:22:49.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:49.076+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:49.085 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:49.080+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:49.928 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:49.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:50.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:50.068+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:50.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:50.068+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:50.664 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:22:50.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:50.660496+0000) 2019-10-18T08:22:51.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:51.060+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:51.066 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:51.060+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:52.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:52.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:52.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:52.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:53.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:53.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:53.068 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:53.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:54.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:54.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:54.106 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:54.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:54.929 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:54.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:22:55.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:55.092+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:55.098 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:55.092+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:56.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:56.052+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:56.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:56.052+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:57.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:57.004+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:57.013 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:57.004+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:57.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:22:57.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:57.689254+0000) 2019-10-18T08:22:57.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:22:57.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:57.906985+0000) 2019-10-18T08:22:58.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:58.032+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:58.040 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:58.032+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:58.938 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:58.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:58.935306+0000) 2019-10-18T08:22:58.939 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:22:58.932+0000 7ff3a7d41700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:22:58.936386+0000) 2019-10-18T08:22:59.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:59.020+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:22:59.028 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:22:59.020+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:22:59.930 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:22:59.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:00.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:00.016+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:00.023 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:00.016+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:00.663 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:23:00.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:00.660654+0000) 2019-10-18T08:23:01.018 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:01.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:01.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:01.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:02.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:02.040+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:02.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:02.040+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:03.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:03.056+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:03.065 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:03.056+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:03.438 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:23:03.432+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:03.435515+0000) 2019-10-18T08:23:04.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:04.048+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:04.060 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:04.048+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:04.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:04.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:05.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:05.100+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:05.109 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:05.100+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:06.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:06.132+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:06.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:06.132+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:07.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:07.152+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:07.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:07.156+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:07.692 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:23:07.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:07.689427+0000) 2019-10-18T08:23:07.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:23:07.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:07.907085+0000) 2019-10-18T08:23:08.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:08.108+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:08.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:08.108+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:09.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:09.120+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:09.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:09.120+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:09.931 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:09.924+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:10.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:10.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:10.145 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:10.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:10.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:23:10.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:10.660826+0000) 2019-10-18T08:23:11.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:11.131+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:11.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:11.131+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:12.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:12.099+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:12.108 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:12.099+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:13.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:13.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:13.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:13.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:13.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:23:13.431+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:13.435617+0000) 2019-10-18T08:23:14.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:14.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:14.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:14.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:14.932 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:14.927+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:15.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:15.167+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:15.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:15.167+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:16.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:16.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:16.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:16.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:17.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:17.179+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:17.188 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:17.179+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:17.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:23:17.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:17.689615+0000) 2019-10-18T08:23:17.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:23:17.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:17.907183+0000) 2019-10-18T08:23:18.192 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:18.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:18.193 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:18.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:19.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:19.139+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:19.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:19.143+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:19.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:19.927+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:20.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:20.095+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:20.102 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:20.095+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:20.664 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:23:20.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:20.661019+0000) 2019-10-18T08:23:21.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:21.051+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:21.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:21.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:22.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:22.090 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:22.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:23.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:23.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:23.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:23.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:23.438 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:23:23.431+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:23.435723+0000) 2019-10-18T08:23:24.036 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:24.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:24.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:24.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:24.934 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:24.927+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:25.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:25.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:25.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:25.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:26.007 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:25.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:26.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:25.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:27.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:26.995+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:27.003 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:26.995+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:27.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:23:27.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:27.689787+0000) 2019-10-18T08:23:27.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:23:27.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:27.907276+0000) 2019-10-18T08:23:27.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:27.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:27.959 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:27.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:28.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:28.947+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:28.956 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:28.947+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:29.935 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:29.927+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:29.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:29.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:29.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:29.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:30.664 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:23:30.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:30.661201+0000) 2019-10-18T08:23:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:30.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:30.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:30.951+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:31.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:31.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:31.998 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:31.991+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:33.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:33.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:33.032 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:33.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:33.440 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:23:33.431+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:33.435835+0000) 2019-10-18T08:23:34.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:34.055+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:34.064 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:34.055+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:34.936 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:34.927+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:35.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:35.047+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:35.057 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:35.051+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:36.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:36.083+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:36.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:36.083+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:37.075 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:37.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:37.076 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:37.067+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:37.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:23:37.683+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:37.689943+0000) 2019-10-18T08:23:37.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:23:37.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:37.907378+0000) 2019-10-18T08:23:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:38.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:38.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:38.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:39.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:39.119+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:39.126 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:39.119+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:39.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:39.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:40.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:40.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:40.131 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:40.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:40.664 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:23:40.655+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:40.661414+0000) 2019-10-18T08:23:41.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:41.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:41.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:41.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:42.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:42.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:42.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:42.091+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:43.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:43.115+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:43.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:43.115+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:43.438 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:23:43.431+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:43.435934+0000) 2019-10-18T08:23:44.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:44.155+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:44.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:44.155+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:44.937 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:44.931+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:45.157 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:45.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:45.158 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:45.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:46.147 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:46.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:46.148 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:46.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:47.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:47.098+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:47.107 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:47.098+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:47.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:23:47.682+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:47.690146+0000) 2019-10-18T08:23:47.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:23:47.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:47.907477+0000) 2019-10-18T08:23:48.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:48.134+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:48.143 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:48.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:49.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:49.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:49.173 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:49.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:49.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:49.930+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:50.150 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:50.142+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:50.152 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:50.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:50.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:23:50.654+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:50.661603+0000) 2019-10-18T08:23:51.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:51.182+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:51.191 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:51.182+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:52.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:52.194+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:52.204 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:52.194+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:53.200 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:53.190+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:53.201 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:53.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:53.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:23:53.430+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:53.436033+0000) 2019-10-18T08:23:54.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:54.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:54.219 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:54.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:54.939 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:54.930+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:23:55.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:55.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:55.195 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:55.186+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:56.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:56.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:56.175 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:56.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:57.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:57.165 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:57.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:57.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:23:57.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:57.690344+0000) 2019-10-18T08:23:57.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:23:57.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:23:57.907575+0000) 2019-10-18T08:23:58.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:58.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:58.130 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:58.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:59.170 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:59.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:23:59.174 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:23:59.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:23:59.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:23:59.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:00.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:00.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:00.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:00.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:00.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:24:00.654+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:00.661820+0000) 2019-10-18T08:24:01.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:01.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:01.135 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:01.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:02.114 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:02.102+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:02.115 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:02.102+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:03.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:03.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:03.149 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:03.142+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:03.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:03.430+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:03.436137+0000) 2019-10-18T08:24:04.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:04.138+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:04.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:04.138+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:04.941 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:04.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:05.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:05.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:05.123 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:05.114+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:06.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:06.150+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:06.162 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:06.150+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:07.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:07.174+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:07.186 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:07.178+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:07.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:24:07.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:07.690523+0000) 2019-10-18T08:24:07.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:24:07.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:07.907677+0000) 2019-10-18T08:24:08.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:08.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:08.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:08.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:09.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:09.258+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:09.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:09.262+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:09.942 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:09.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:10.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:10.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:10.229 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:10.222+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:10.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:24:10.654+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:10.662035+0000) 2019-10-18T08:24:11.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:11.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:11.277 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:11.270+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:12.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:12.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:12.276 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:12.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:13.274 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:13.266+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:13.275 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:13.266+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:13.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:13.430+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:13.436238+0000) 2019-10-18T08:24:14.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:14.230+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:14.240 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:14.230+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:14.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:14.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:15.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:15.282+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:15.291 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:15.282+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:16.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:16.314+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:16.324 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:16.314+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:17.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:17.334+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:17.345 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:17.334+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:17.693 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:24:17.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:17.690715+0000) 2019-10-18T08:24:17.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:24:17.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:17.907777+0000) 2019-10-18T08:24:18.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:18.322+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:18.331 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:18.322+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:19.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:19.306+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:19.316 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:19.306+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:19.943 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:19.934+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:20.348 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:20.337+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:20.349 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:20.337+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:20.665 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:24:20.658+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:20.662220+0000) 2019-10-18T08:24:21.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:21.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:21.335 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:21.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:22.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:22.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:22.381 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:22.373+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:23.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:23.325+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:23.337 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:23.325+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:23.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:23.429+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:23.436338+0000) 2019-10-18T08:24:24.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:24.333+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:24.343 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:24.333+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:24.944 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:24.933+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:25.392 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:25.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:25.393 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:25.385+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:26.435 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:26.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:26.436 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:26.425+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:27.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:27.389+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:27.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:27.393+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:27.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:24:27.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:27.690901+0000) 2019-10-18T08:24:27.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:24:27.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:27.907878+0000) 2019-10-18T08:24:28.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:28.409+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:28.419 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:28.409+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:29.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:29.381+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:29.391 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:29.381+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:29.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:29.937+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:30.373 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:30.361+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:30.374 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:30.361+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:30.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:24:30.658+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:30.662412+0000) 2019-10-18T08:24:31.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:31.349+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:31.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:31.349+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:32.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:32.369+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:32.378 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:32.369+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:33.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:33.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:33.429 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:33.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:33.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:33.429+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:33.436428+0000) 2019-10-18T08:24:34.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:34.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:34.423 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:34.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:34.946 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:34.937+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:35.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:35.397+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:35.408 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:35.397+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:36.437 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:36.425+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:36.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:36.429+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:37.432 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:37.421+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:37.433 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:37.421+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:37.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:24:37.686+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:37.691112+0000) 2019-10-18T08:24:37.910 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:24:37.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:37.907979+0000) 2019-10-18T08:24:38.410 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:38.401+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:38.411 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:38.401+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:39.417+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:39.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:39.417+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:39.947 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:39.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:40.420 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:40.413+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:40.421 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:40.413+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:40.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:24:40.658+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:40.662628+0000) 2019-10-18T08:24:41.464 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:41.457+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:41.465 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:41.457+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:42.444 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:42.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:42.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:42.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:43.439 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:43.433+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:43.436519+0000) 2019-10-18T08:24:43.440 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:43.437+0000 7ff3a8542700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:43.437711+0000) 2019-10-18T08:24:43.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:43.461+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:43.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:43.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:44.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:44.513+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:44.519 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:44.513+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:44.948 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:44.941+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:45.558 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:45.553+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:45.559 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:45.553+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:46.563 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:46.557+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:46.564 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:46.557+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:47.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:47.581+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:47.586 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:47.581+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:47.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:24:47.690+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:47.691298+0000) 2019-10-18T08:24:47.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:24:47.905+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:47.908080+0000) 2019-10-18T08:24:47.939 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:47.933+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:47.936735+0000) 2019-10-18T08:24:48.568 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:48.561+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:48.570 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:48.565+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:49.534 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:49.529+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:49.535 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:49.529+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:49.949 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:49.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:50.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:50.481+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:50.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:50.481+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:50.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:24:50.662+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:50.662819+0000) 2019-10-18T08:24:51.449 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:51.437+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:51.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:51.437+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:52.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:52.465+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:52.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:52.465+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:53.477+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:53.484 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:53.477+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:54.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:54.433+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:54.439 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:54.433+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:54.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:54.945+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:24:55.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:55.440+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:55.447 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:55.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:56.468 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:56.460+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:56.469 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:56.460+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:57.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:57.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:57.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:57.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:57.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:24:57.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:57.691507+0000) 2019-10-18T08:24:57.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:24:57.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:57.908182+0000) 2019-10-18T08:24:57.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:24:57.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:24:57.936830+0000) 2019-10-18T08:24:58.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:58.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:58.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:58.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:59.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:59.368+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:24:59.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:24:59.368+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:24:59.950 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:24:59.944+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:00.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:00.360+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:00.367 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:00.360+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:00.669 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:25:00.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:00.663031+0000) 2019-10-18T08:25:01.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:01.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:01.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:01.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:02.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:02.416+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:02.424 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:02.416+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:03.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:03.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:03.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:03.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:04.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:04.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:04.443 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:04.436+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:04.951 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:04.944+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:05.452 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:05.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:05.453 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:05.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:06.491 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:06.484+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:06.492 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:06.484+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:07.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:07.516+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:07.524 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:07.516+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:07.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:25:07.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:07.691694+0000) 2019-10-18T08:25:07.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:25:07.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:07.908283+0000) 2019-10-18T08:25:07.939 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:25:07.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:07.936932+0000) 2019-10-18T08:25:08.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:08.480+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:08.487 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:08.480+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:09.459 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:09.452+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:09.460 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:09.452+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:09.952 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:09.944+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:10.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:10.432+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:10.438 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:10.432+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:10.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:25:10.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:10.663237+0000) 2019-10-18T08:25:11.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:11.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:11.427 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:11.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:12.454 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:12.448+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:12.455 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:12.448+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:13.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:13.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:13.477 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:13.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:14.450 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:14.444+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:14.451 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:14.444+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:14.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:14.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:15.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:15.468+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:15.476 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:15.468+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:16.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:16.436+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:16.445 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:16.440+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:17.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:17.408+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:17.415 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:17.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:17.694 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:25:17.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:17.691880+0000) 2019-10-18T08:25:17.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:25:17.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:17.908382+0000) 2019-10-18T08:25:17.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:25:17.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:17.937028+0000) 2019-10-18T08:25:18.397 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:18.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:18.398 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:18.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:19.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:19.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:19.401 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:19.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:19.953 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:19.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:20.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:20.388+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:20.395 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:20.388+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:20.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:25:20.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:20.663423+0000) 2019-10-18T08:25:21.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:21.380+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:21.388 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:21.380+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:22.379 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:22.372+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:22.380 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:22.372+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:23.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:23.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:23.385 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:23.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:24.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:24.352+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:24.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:24.352+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:24.954 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:24.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:25.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:25.348+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:25.355 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:25.348+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:26.399 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:26.392+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:26.400 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:26.392+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:27.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:27.404+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:27.413 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:27.408+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:27.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:25:27.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:27.692069+0000) 2019-10-18T08:25:27.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:25:27.904+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:27.908483+0000) 2019-10-18T08:25:27.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:25:27.932+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:27.937123+0000) 2019-10-18T08:25:28.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:28.420+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:28.428 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:28.420+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:29.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:29.376+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:29.382 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:29.376+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:29.955 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:29.948+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:30.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:30.383 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:30.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:30.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:25:30.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:30.663619+0000) 2019-10-18T08:25:31.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:31.375+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:31.384 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:31.375+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:32.336 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:32.327+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:32.338 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:32.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:33.356 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:33.347+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:33.357 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:33.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:34.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:34.351+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:34.358 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:34.351+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:34.956 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:34.947+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:35.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:35.315+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:35.323 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:35.315+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:36.303 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:36.295+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:36.304 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:36.295+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:37.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:37.263+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:37.272 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:37.263+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:37.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:25:37.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:37.692214+0000) 2019-10-18T08:25:37.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:25:37.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:37.908583+0000) 2019-10-18T08:25:37.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:25:37.931+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:37.937223+0000) 2019-10-18T08:25:38.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:38.271+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:38.280 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:38.271+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:39.264 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:39.255+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:39.265 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:39.255+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:39.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:39.951+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:40.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:40.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:40.223 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:40.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:40.666 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:25:40.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:40.663810+0000) 2019-10-18T08:25:41.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:41.215+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:41.228 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:41.219+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:42.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:42.235+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:42.245 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:42.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:43.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:43.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:43.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:43.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:44.220 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:44.211+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:44.222 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:44.215+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:44.965 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:44.951+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:45.269 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:45.259+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:45.270 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:45.259+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:46.288 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:46.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:46.289 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:46.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:47.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:47.331+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:47.340 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:47.331+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:47.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:25:47.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:47.692424+0000) 2019-10-18T08:25:47.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:25:47.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:47.908685+0000) 2019-10-18T08:25:47.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:25:47.931+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:47.937319+0000) 2019-10-18T08:25:48.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:48.307+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:48.315 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:48.307+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:49.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:49.279+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:49.287 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:49.279+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:49.958 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:49.951+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:50.247 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:50.239+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:50.248 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:50.239+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:50.667 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:25:50.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:50.664026+0000) 2019-10-18T08:25:51.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:51.203+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:51.212 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:51.203+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:52.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:52.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:52.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:52.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:53.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:53.195+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:53.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:53.199+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:54.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:54.183+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:54.190 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:54.183+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:54.959 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:54.951+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:25:55.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:55.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:55.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:55.163+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:56.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:56.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:56.146 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:56.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:57.133 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:57.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:57.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:57.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:57.695 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:25:57.689+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:57.692616+0000) 2019-10-18T08:25:57.911 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:25:57.903+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:57.908784+0000) 2019-10-18T08:25:57.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:25:57.931+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:25:57.937415+0000) 2019-10-18T08:25:58.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:58.123+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:58.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:58.123+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:59.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:59.111+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:25:59.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:25:59.111+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:25:59.960 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:25:59.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:00.168 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:00.159+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:00.169 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:00.159+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:00.667 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:26:00.661+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:00.664213+0000) 2019-10-18T08:26:01.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:01.191+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:01.205 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:01.191+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:02.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:02.227+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:02.237 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:02.231+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:03.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:03.231+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:03.241 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:03.235+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:04.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:04.207+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:04.214 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:04.207+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:04.961 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:04.955+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:05.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:05.218+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:05.226 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:05.218+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:06.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:06.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:06.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:06.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:07.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:07.126+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:07.139 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:07.126+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:07.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:26:07.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:07.692800+0000) 2019-10-18T08:26:07.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:26:07.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:07.908885+0000) 2019-10-18T08:26:07.951 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:26:07.934+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:07.937523+0000) 2019-10-18T08:26:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:08.114+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:08.125 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:08.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:09.086 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:09.078+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:09.088 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:09.078+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:09.962 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:09.954+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:10.049 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:10.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:10.050 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:10.042+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:10.667 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:26:10.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:10.664399+0000) 2019-10-18T08:26:11.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:11.026+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:11.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:11.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:12.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:12.030+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:12.039 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:12.030+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:13.024 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:13.014+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:13.026 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:13.018+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:14.070 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:14.062+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:14.071 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:14.062+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:14.963 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:14.954+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:15.038+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:15.047 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:15.038+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:16.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:16.082+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:16.093 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:16.082+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:17.096 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:17.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:17.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:17.086+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:17.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:26:17.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:17.692990+0000) 2019-10-18T08:26:17.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:26:17.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:17.908988+0000) 2019-10-18T08:26:17.942 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:26:17.934+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:17.937623+0000) 2019-10-18T08:26:18.120 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:18.110+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:18.122 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:18.110+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:19.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:19.122+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:19.134 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:19.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:19.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:19.954+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:20.176 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:20.166+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:20.177 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:20.166+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:20.668 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:26:20.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:20.664613+0000) 2019-10-18T08:26:21.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:21.154+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:21.164 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:21.154+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:22.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:22.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:22.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:22.122+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:23.141 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:23.130+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:23.142 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:23.134+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:24.180 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:24.170+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:24.181 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:24.170+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:24.964 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:24.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:25.216 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:25.206+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:25.217 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:25.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:26.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:26.210+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:26.218 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:26.210+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:27.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:27.186+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:27.198 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:27.190+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:27.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:26:27.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:27.693195+0000) 2019-10-18T08:26:27.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:26:27.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:27.909086+0000) 2019-10-18T08:26:27.940 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:26:27.934+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:27.937718+0000) 2019-10-18T08:26:28.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:28.158+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:28.166 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:28.158+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:29.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:29.118+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:29.132 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:29.118+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:29.969 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:29.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:30.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:30.074+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:30.084 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:30.074+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:30.667 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:26:30.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:30.664792+0000) 2019-10-18T08:26:31.097 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:31.086+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:31.121 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:31.090+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:32.058 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:32.050+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:32.059 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:32.050+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:33.011 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:33.002+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:33.012 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:33.002+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:34.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:34.010+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:34.019 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:34.010+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:34.966 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:34.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:35.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:34.986+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:35.001 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:34.990+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:35.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:35.966+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:35.975 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:35.966+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:36.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:36.946+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:36.957 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:36.946+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:37.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:26:37.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:37.693382+0000) 2019-10-18T08:26:37.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:37.902+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:37.911 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:37.902+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:37.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:26:37.902+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:37.909188+0000) 2019-10-18T08:26:37.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:26:37.934+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:37.937820+0000) 2019-10-18T08:26:38.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:38.894+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:38.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:38.894+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:39.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:39.874+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:39.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:39.874+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:39.967 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:39.958+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:40.668 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:26:40.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:40.664972+0000) 2019-10-18T08:26:40.891 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:40.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:40.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:40.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:41.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:41.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:41.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:41.845+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:42.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:42.893+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:42.908 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:42.893+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:43.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:43.913+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:43.923 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:43.913+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:44.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:44.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:44.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:44.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:44.968 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:44.957+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:45.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:45.849+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:45.859 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:45.849+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:46.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:46.809+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:46.818 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:46.809+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:47.696 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:26:47.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:47.693560+0000) 2019-10-18T08:26:47.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:47.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:47.840 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:47.829+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:47.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:26:47.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:47.909291+0000) 2019-10-18T08:26:47.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:26:47.933+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:47.937918+0000) 2019-10-18T08:26:48.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:48.829+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:48.842 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:48.833+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:49.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:49.877+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:49.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:49.877+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:49.969 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:49.961+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:50.668 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:26:50.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:50.665154+0000) 2019-10-18T08:26:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:50.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:50.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:50.889+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:51.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:51.861+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:51.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:51.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:52.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:52.885+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:52.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:52.885+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:53.852 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:53.841+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:53.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:53.841+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:54.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:54.837+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:54.846 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:54.837+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:54.970 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:54.961+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:26:55.875 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:55.865+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:55.876 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:55.865+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:56.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:56.881+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:56.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:56.881+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:57.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:26:57.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:57.693738+0000) 2019-10-18T08:26:57.906 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:57.897+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:57.907 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:57.897+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:57.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:26:57.901+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:57.909393+0000) 2019-10-18T08:26:57.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:26:57.933+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:26:57.938012+0000) 2019-10-18T08:26:58.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:58.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:58.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:58.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:59.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:59.941+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:26:59.951 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:26:59.941+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:26:59.971 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:26:59.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:00.669 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:27:00.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:00.665297+0000) 2019-10-18T08:27:00.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:00.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:00.962 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:00.957+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:02.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:02.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:02.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:02.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:03.037 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:03.029+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:03.038 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:03.033+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:04.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:04.041+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:04.048 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:04.041+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:04.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:04.965+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:05.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:05.001+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:05.010 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:05.005+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:05.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:05.985+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:05.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:05.985+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:06.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:06.969+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:06.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:06.969+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:07.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:27:07.688+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:07.693923+0000) 2019-10-18T08:27:07.917 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:27:07.909+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:07.909461+0000) 2019-10-18T08:27:07.942 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:27:07.937+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:07.938113+0000) 2019-10-18T08:27:08.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:07.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:08.000 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:07.993+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:09.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:08.993+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:09.002 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:08.997+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:09.972 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:09.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:10.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:10.009+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:10.015 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:10.009+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:10.668 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:27:10.660+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:10.665475+0000) 2019-10-18T08:27:10.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:10.977+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:10.986 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:10.981+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:11.933+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:11.939 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:11.933+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:12.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:12.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:12.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:12.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:13.915 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:13.909+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:13.916 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:13.909+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:14.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:14.953+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:14.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:14.953+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:14.973 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:14.969+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:15.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:15.952+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:15.958 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:15.952+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:16.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:16.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:16.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:16.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:17.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:27:17.687+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:17.694110+0000) 2019-10-18T08:27:17.896 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:17.888+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:17.897 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:17.892+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:17.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:27:17.908+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:17.909559+0000) 2019-10-18T08:27:17.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:27:17.936+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:17.938205+0000) 2019-10-18T08:27:18.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:18.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:18.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:18.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:19.947 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:19.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:19.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:19.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:19.974 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:19.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:20.668 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:27:20.659+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:20.665670+0000) 2019-10-18T08:27:20.992 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:20.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:20.993 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:20.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:22.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:22.012+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:22.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:22.012+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:22.995 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:22.988+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:22.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:22.988+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:23.984+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:23.991 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:23.984+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:24.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:24.940+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:24.948 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:24.940+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:24.975 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:24.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:25.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:25.896+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:25.904 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:25.896+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:26.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:26.912+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:26.920 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:26.912+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:27.699 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:27:27.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:27.694310+0000) 2019-10-18T08:27:27.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:27:27.908+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:27.909659+0000) 2019-10-18T08:27:27.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:27.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:27.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:27.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:27.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:27:27.936+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:27.938299+0000) 2019-10-18T08:27:28.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:28.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:28.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:28.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:29.932 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:29.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:29.933 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:29.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:29.976 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:29.968+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:30.669 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:27:30.659+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:30.665853+0000) 2019-10-18T08:27:30.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:30.924+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:30.931 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:30.924+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:31.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:31.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:31.922 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:31.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:32.892 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:32.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:32.893 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:32.888+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:33.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:33.844+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:33.853 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:33.844+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:34.815 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:34.808+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:34.816 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:34.808+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:34.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:34.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:35.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:35.776+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:35.783 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:35.776+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:36.764 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:36.756+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:36.765 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:36.756+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:37.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:27:37.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:37.694461+0000) 2019-10-18T08:27:37.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:37.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:37.797 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:37.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:37.912 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:27:37.908+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:37.909758+0000) 2019-10-18T08:27:37.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:27:37.936+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:37.938396+0000) 2019-10-18T08:27:38.786 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:38.780+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:38.787 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:38.780+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:39.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:39.788+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:39.794 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:39.788+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:39.977 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:39.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:40.675 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:27:40.659+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:40.666036+0000) 2019-10-18T08:27:40.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:40.760+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:40.768 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:40.760+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:41.770 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:41.764+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:41.771 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:41.764+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:42.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:42.792+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:42.800 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:42.792+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:43.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:43.824+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:43.834 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:43.828+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:44.823 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:44.816+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:44.824 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:44.816+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:44.978 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:44.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:45.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:45.864+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:45.873 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:45.864+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:46.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:46.884+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:46.890 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:46.884+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:47.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:27:47.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:47.694628+0000) 2019-10-18T08:27:47.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:47.904+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:47.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:47.904+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:47.913 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:27:47.908+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:47.909856+0000) 2019-10-18T08:27:47.941 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:27:47.936+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:47.938490+0000) 2019-10-18T08:27:48.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:48.916+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:48.924 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:48.916+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:49.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:49.944+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:49.953 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:49.948+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:49.979 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:49.972+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:50.669 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:27:50.663+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:50.666259+0000) 2019-10-18T08:27:50.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:50.899+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:50.912 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:50.903+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:51.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:51.867+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:51.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:51.867+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:52.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:52.895+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:52.905 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:52.895+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:53.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:53.891+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:53.898 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:53.891+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:54.882 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:54.875+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:54.883 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:54.875+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:54.980 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:54.971+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:27:55.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:55.863+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:55.872 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:55.863+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:56.888 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:56.879+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:56.889 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:56.883+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:57.697 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:27:57.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:57.694793+0000) 2019-10-18T08:27:57.855 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:57.847+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:57.856 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:57.847+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:57.914 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:27:57.907+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:57.909954+0000) 2019-10-18T08:27:57.942 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:27:57.935+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:27:57.938588+0000) 2019-10-18T08:27:58.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:58.887+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:58.895 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:58.887+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:59.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:59.931+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:27:59.938 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:27:59.931+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:27:59.981 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:27:59.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:28:00.670 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:28:00.663+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:28:00.666446+0000) 2019-10-18T08:28:00.960 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:00.951+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:00.961 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:00.955+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:01.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:01.963+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:01.974 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:01.963+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:03.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:02.999+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:03.008 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:02.999+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:04.034 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:04.027+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:04.035 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:04.027+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:04.982 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:28:04.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:28:05.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:05.031+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:05.041 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:05.031+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:05.987+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:05.996 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:05.987+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:07.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:07.011+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:07.021 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:07.015+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:07.698 INFO:tasks.ceph.osd.5.smithi204.stderr:2019-10-18T08:28:07.691+0000 7f4509f3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:28:07.694977+0000) 2019-10-18T08:28:07.913 INFO:tasks.ceph.osd.0.smithi092.stderr:2019-10-18T08:28:07.907+0000 7fb6280a7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:28:07.910052+0000) 2019-10-18T08:28:07.942 INFO:tasks.ceph.osd.3.smithi092.stderr:2019-10-18T08:28:07.935+0000 7ff38d2ba700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:28:07.938687+0000) 2019-10-18T08:28:08.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:08.023+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:08.030 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:08.023+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:09.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:09.067+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:09.081 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:09.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:09.985 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:28:09.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:28:10.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:10.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:10.080 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:10.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:10.670 INFO:tasks.ceph.osd.7.smithi204.stderr:2019-10-18T08:28:10.663+0000 7f5506c84700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2019-10-18T07:28:10.666667+0000) 2019-10-18T08:28:11.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:11.071+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:11.079 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:11.071+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:12.094 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:12.087+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:12.095 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:12.087+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:13.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:13.135+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:13.144 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:13.135+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:14.136 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:14.127+0000 7fd46a48c700 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.92:6830 osd.2 ever on either front or back, first ping sent 2019-10-18T08:18:27.994791+0000 (oldest deadline 2019-10-18T08:18:47.994791+0000) 2019-10-18T08:28:14.137 INFO:tasks.ceph.osd.1.smithi092.stderr:2019-10-18T08:28:14.127+0000 7fd46a48c700 -1 osd.1 29 get_health_metrics reporting 612 slow ops, oldest is osd_op(client.4423.0:17 2.f 2.49900b6f (undecoded) ondisk+retry+write+known_if_redirected e29) 2019-10-18T08:28:14.983 INFO:tasks.ceph.mon.a.smithi092.stderr:2019-10-18T08:28:14.975+0000 7fc49b0a4700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.2 [v2:172.21.15.92:6822/13092,v1:172.21.15.92:6824/13092] for 29sec e24 v24) 2019-10-18T08:28:15.101 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-10-18T08:28:15.101 DEBUG:teuthology.task.console_log:Killing console logger for smithi204 2019-10-18T08:28:15.101 DEBUG:teuthology.task.console_log:Killing console logger for smithi092 2019-10-18T08:28:15.102 DEBUG:teuthology.task.console_log:Killing console logger for smithi204 2019-10-18T08:28:15.102 DEBUG:teuthology.task.console_log:Killing console logger for smithi092 2019-10-18T08:28:15.102 DEBUG:teuthology.exit:Finished running handlers